Oct 31 19:13:45.631687 ip-10-0-128-164 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 19:13:45.631699 ip-10-0-128-164 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 19:13:45.631708 ip-10-0-128-164 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 19:13:45.632035 ip-10-0-128-164 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 19:13:55.755863 ip-10-0-128-164 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 19:13:55.755885 ip-10-0-128-164 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a5f8d881850448f38245e8aa8a525751 -- Oct 31 19:15:55.320189 ip-10-0-128-164 systemd[1]: Starting Kubernetes Kubelet... Oct 31 19:15:55.775323 ip-10-0-128-164 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:15:55.775323 ip-10-0-128-164 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 19:15:55.775323 ip-10-0-128-164 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:15:55.775323 ip-10-0-128-164 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:15:55.778624 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778546 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:55.778624 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778624 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778629 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778632 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778635 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778638 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778641 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778644 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778647 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778649 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778653 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778655 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778660 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778665 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778668 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778671 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778674 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778677 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778679 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778682 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:55.778687 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778684 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778687 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778690 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778693 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778696 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778699 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778701 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778704 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778706 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778709 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778712 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778714 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778717 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778720 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778722 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778725 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778727 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778730 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778732 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:55.779126 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778735 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778737 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778740 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778742 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778745 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778747 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778750 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778752 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778755 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778758 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778760 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778764 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778766 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778769 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778771 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778775 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778778 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778780 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778783 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778785 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:55.779584 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778788 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778790 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778793 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778795 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778798 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778800 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778802 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778805 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778807 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778810 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778812 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778815 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778818 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778820 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778823 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778825 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778827 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778830 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778833 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778835 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:55.780059 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778838 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778840 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778843 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778845 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778848 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778850 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778853 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778856 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778859 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778861 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778864 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778867 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778869 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778872 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778874 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778890 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778894 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778898 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778901 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778904 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:55.780557 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778907 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778909 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778912 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778914 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778917 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778919 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.778922 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779704 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779711 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779714 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779717 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779720 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779723 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779726 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779728 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779731 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779734 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779736 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779739 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779743 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:55.781419 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779746 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779748 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779751 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779754 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779756 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779759 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779762 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779764 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779767 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779769 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779772 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779775 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779777 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779780 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779783 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779785 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779788 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779791 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779793 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779796 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:55.782119 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779798 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779801 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779803 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779806 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779809 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779812 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779814 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779817 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779819 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779822 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779825 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779828 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779831 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779834 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779837 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779839 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779841 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779844 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779847 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:55.782582 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779849 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779851 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779855 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779858 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779861 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779863 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779866 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779868 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779870 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779873 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779876 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779893 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779897 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779901 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779905 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779910 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779912 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779918 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779922 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:55.783044 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779926 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779929 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779932 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779935 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779938 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779941 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779944 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779946 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779949 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779951 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779954 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779957 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779959 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779962 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779964 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779967 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779970 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779973 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779975 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779978 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:55.783516 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779980 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779983 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779985 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779988 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779990 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779993 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.779998 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.780000 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.780003 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.780006 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.780008 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.780011 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.780014 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.780016 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.780019 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781203 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781214 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781219 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781224 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781234 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:55.784007 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781239 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781243 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781247 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781253 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781259 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781263 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781267 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781270 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781273 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781276 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781279 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781281 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781284 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781287 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781290 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781292 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781295 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781300 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781302 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:55.784485 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781305 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781307 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781310 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781313 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781317 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781320 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781323 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781325 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781328 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781330 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781333 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781341 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781345 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781349 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781353 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781357 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781361 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781365 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781369 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781374 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:55.784958 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781378 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781383 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781387 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781395 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781399 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781404 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781408 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781411 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781415 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781419 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781423 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781428 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781432 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781436 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781440 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781445 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781454 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781459 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781463 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781468 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:55.785440 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781472 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781476 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781480 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781485 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781489 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781494 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781500 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781504 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781513 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781517 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781522 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781526 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781530 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781534 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781538 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781542 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781546 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781550 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781554 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781558 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:55.785931 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781562 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781570 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781574 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781578 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781582 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781586 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781591 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781644 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781791 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781799 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781802 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781805 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781808 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781811 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781814 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781816 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781819 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781822 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781824 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:55.786411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781827 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781829 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.781835 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783674 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783686 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783694 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783699 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783704 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783707 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783712 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783716 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783720 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783723 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783726 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783730 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783734 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783737 2578 flags.go:64] FLAG: --cgroup-root="" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783741 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783744 2578 flags.go:64] FLAG: --client-ca-file="" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783747 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783751 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783755 2578 flags.go:64] FLAG: --cluster-domain="" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783758 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 19:15:55.786845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783762 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783765 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783769 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783773 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783777 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783780 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783783 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783786 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783790 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783793 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783796 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783799 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783803 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783807 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783809 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783812 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783816 2578 flags.go:64] FLAG: --enable-server="true" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783819 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783823 2578 flags.go:64] FLAG: --event-burst="100" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783826 2578 flags.go:64] FLAG: --event-qps="50" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783829 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783832 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783835 2578 flags.go:64] FLAG: --eviction-hard="" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783839 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 19:15:55.787407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783843 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783846 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783850 2578 flags.go:64] FLAG: --eviction-soft="" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783853 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783856 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783859 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783862 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783865 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783868 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783871 2578 flags.go:64] FLAG: --feature-gates="" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783875 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783893 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783897 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783901 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783904 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783907 2578 flags.go:64] FLAG: --help="false" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783910 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783913 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783916 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783919 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783922 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783926 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783929 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 19:15:55.788022 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783931 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783934 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783937 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783940 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783944 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783947 2578 flags.go:64] FLAG: --kube-reserved="" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783950 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783953 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783956 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783960 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783964 2578 flags.go:64] FLAG: --lock-file="" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783966 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783970 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783973 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783979 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783983 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783986 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783989 2578 flags.go:64] FLAG: --logging-format="text" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783992 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783996 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.783999 2578 flags.go:64] FLAG: --manifest-url="" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784002 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784007 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784010 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784014 2578 flags.go:64] FLAG: --max-pods="110" Oct 31 19:15:55.788586 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784018 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784021 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784024 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784027 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784030 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784033 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784036 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784045 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784048 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784051 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784054 2578 flags.go:64] FLAG: --pod-cidr="" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784057 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784060 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784063 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784066 2578 flags.go:64] FLAG: --port="10250" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784069 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784072 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0dfaf2352953a1145" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784075 2578 flags.go:64] FLAG: --qos-reserved="" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784079 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784082 2578 flags.go:64] FLAG: --register-node="true" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784085 2578 flags.go:64] FLAG: --register-with-taints="" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784089 2578 flags.go:64] FLAG: --registry-burst="10" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784092 2578 flags.go:64] FLAG: --registry-qps="5" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784095 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784098 2578 flags.go:64] FLAG: --reserved-memory="" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784102 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 19:15:55.789202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784105 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784108 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784111 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784114 2578 flags.go:64] FLAG: --runonce="false" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784117 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784120 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784123 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784126 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784129 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784132 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784135 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784138 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784141 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784144 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784147 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784150 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784153 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784156 2578 flags.go:64] FLAG: --system-cgroups="" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784158 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784164 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784167 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784170 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784175 2578 flags.go:64] FLAG: --tls-min-version="" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784178 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784181 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 19:15:55.789835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784185 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784188 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784191 2578 flags.go:64] FLAG: --v="2" Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784196 2578 flags.go:64] FLAG: --version="false" Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784200 2578 flags.go:64] FLAG: --vmodule="" Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784205 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784208 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784344 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784348 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784351 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784354 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784357 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784359 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784362 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784365 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784369 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784372 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784375 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784377 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784380 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784382 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:55.790498 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784385 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784388 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784390 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784393 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784395 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784398 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784401 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784403 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784408 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784411 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784414 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784417 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784422 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784426 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784429 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784432 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784435 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784437 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784440 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:55.791006 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784443 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784445 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784448 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784451 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784453 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784456 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784459 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784461 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784464 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784466 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784469 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784485 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784488 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784491 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784494 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784496 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784499 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784502 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784504 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784507 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:55.791492 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784510 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784514 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784517 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784519 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784522 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784525 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784528 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784531 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784533 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784536 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784538 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784541 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784543 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784546 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784548 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784550 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784553 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784555 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784558 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:55.791975 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784560 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784563 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784565 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784567 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784570 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784573 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784575 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784577 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784580 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784582 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784585 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784587 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784590 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784592 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784595 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784598 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784600 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784603 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784605 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784609 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:55.792522 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784612 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784614 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784617 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784619 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784622 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784624 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784627 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784629 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784631 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784634 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784636 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784639 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784641 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784644 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.784670 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784711 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:55.792995 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784716 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784718 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784721 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784724 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784726 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784729 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784731 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784734 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784736 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784738 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784741 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784745 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784748 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784750 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784753 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784757 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784760 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784763 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784765 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:55.793361 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784768 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784771 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784773 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784776 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784778 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784780 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784783 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784785 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784788 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784790 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784793 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784796 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784798 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784800 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784803 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784805 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784808 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784810 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784813 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784815 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:55.793796 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784817 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784820 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784822 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784825 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784828 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784831 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784833 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784836 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784840 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784842 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784845 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784848 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784850 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784853 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784855 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784858 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784860 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784863 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784865 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784868 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:55.794333 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784870 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784873 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784875 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784895 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784898 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784902 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784904 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784907 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784909 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784912 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784914 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784918 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784922 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784924 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784927 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784929 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784935 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784937 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784940 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:55.794799 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784943 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784947 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784949 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784952 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784955 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784958 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784961 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784963 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784966 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784969 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784972 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784976 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784980 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784983 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784986 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784988 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784991 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784993 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784996 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:55.795303 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.784999 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.785001 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.785004 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.785006 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.785009 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.785012 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.785015 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.785017 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.792914 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.792933 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.792949 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.792954 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793012 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793017 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793021 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793025 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793028 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793031 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793034 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793037 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:55.795767 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793040 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793043 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793045 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793048 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793050 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793053 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793056 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793059 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793061 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793064 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793066 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793069 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793071 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793074 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793076 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793079 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793081 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793084 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793086 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793089 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:55.796255 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793092 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793097 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793099 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793102 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793104 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793107 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793110 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793112 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793115 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793117 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793120 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793122 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793125 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793127 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793129 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793132 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793134 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793137 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793139 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793142 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:55.796734 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793144 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793147 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793149 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793152 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793154 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793156 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793159 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793161 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793164 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793166 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793169 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793172 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793174 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793177 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793180 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793183 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793185 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793188 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793191 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:55.797180 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793193 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793196 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793198 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793201 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793203 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793205 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793208 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793210 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793213 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793215 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793217 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793220 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793222 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793225 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793227 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793230 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793232 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793235 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793238 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:55.797616 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793243 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793246 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793249 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793252 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793255 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793258 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793260 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793263 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793265 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793269 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793272 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793274 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793277 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793279 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793282 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793285 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793287 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793290 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793292 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:55.798112 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793295 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.793322 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793458 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793462 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793465 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793469 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793472 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793475 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793478 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793481 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793483 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793486 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793489 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793491 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793494 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793496 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:55.798552 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793499 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793502 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793504 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793507 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793509 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793512 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793514 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793518 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793520 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793524 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793527 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793530 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793533 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793537 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793541 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793544 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793547 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793549 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793552 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:55.798941 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793554 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793557 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793559 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793562 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793564 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793567 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793569 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793572 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793575 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793577 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793580 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793582 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793584 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793587 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793589 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793592 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793594 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793597 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793599 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793602 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:55.799411 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793604 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793607 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793610 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793612 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793615 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793618 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793620 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793623 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793625 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793627 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793630 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793633 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793635 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793637 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793640 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793642 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793645 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793647 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793650 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793652 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:55.799898 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793655 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793658 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793660 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793662 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793665 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793667 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793670 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793672 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793675 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793677 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793679 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793682 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793684 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793687 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793690 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793693 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793695 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793700 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793703 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793705 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:55.800376 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793708 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793710 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793712 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793715 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793717 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793720 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793722 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793725 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793728 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793730 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793732 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793735 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793738 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.793760 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793791 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:55.800835 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793796 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793799 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793802 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793805 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793807 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793810 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793812 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793815 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793817 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793820 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793823 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793825 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793828 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793831 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793834 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793837 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793839 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793842 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793845 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793847 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:55.801193 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793850 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793852 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793855 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793857 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793860 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793865 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793868 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793871 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793873 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793876 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793895 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793900 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793902 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793905 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793908 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793910 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793913 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793915 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793918 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:55.801632 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793920 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793922 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793925 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793928 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793930 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793933 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793936 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793940 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793943 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793945 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793948 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793950 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793953 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793955 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793958 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793961 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793963 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793966 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793970 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:55.802173 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793972 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793975 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793977 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793980 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793983 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793985 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793988 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793990 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793993 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.793996 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794000 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794003 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794006 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794009 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794014 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794018 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794020 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794023 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794026 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:55.802617 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794028 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794032 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794035 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794037 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794040 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794042 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794044 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794047 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794049 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794052 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794054 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794057 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794060 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794063 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794065 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794068 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794070 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794073 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794075 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794078 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:55.803083 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794080 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794083 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794085 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794088 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794090 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794093 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794095 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:55.794098 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.794157 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.797577 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.798574 2578 server.go:1014] "Starting client certificate rotation" Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.798666 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.798708 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.800719 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:15:55.803561 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.801796 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:15:55.815940 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.815921 2578 log.go:25] "Validated CRI v1 runtime API" Oct 31 19:15:55.819893 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.819872 2578 log.go:25] "Validated CRI v1 image API" Oct 31 19:15:55.820773 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.820742 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 19:15:55.825386 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.825359 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7e827645-b8f0-4bd4-8392-b8ef4d590083:/dev/nvme0n1p3 b8ed8738-120f-48db-a50f-a4e84c100ee2:/dev/nvme0n1p4] Oct 31 19:15:55.825469 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.825384 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 19:15:55.832771 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.832662 2578 manager.go:217] Machine: {Timestamp:2026-10-31 19:15:55.831111721 +0000 UTC m=+0.379781861 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126077 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aaf51cf45e047196186a5ab70eab0 SystemUUID:ec2aaf51-cf45-e047-1961-86a5ab70eab0 BootID:a5f8d881-8504-48f3-8245-e8aa8a525751 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f6:a2:2c:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f6:a2:2c:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:a9:71:a8:f4:66 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 19:15:55.832771 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.832769 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 19:15:55.832922 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.832902 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 19:15:55.836093 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.836066 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 19:15:55.836228 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.836095 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-164.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 19:15:55.836866 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.836857 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 19:15:55.836916 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.836868 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 19:15:55.836916 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.836907 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 19:15:55.837002 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.836918 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 19:15:55.838058 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.838048 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 19:15:55.842154 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.842142 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 19:15:55.842278 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.842269 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 19:15:55.842940 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.842931 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 31 19:15:55.842968 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.842944 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 19:15:55.842968 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.842954 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 19:15:55.842968 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.842964 2578 kubelet.go:405] "Adding apiserver pod source" Oct 31 19:15:55.843044 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.842971 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 19:15:55.843355 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.843342 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 19:15:55.846003 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.845986 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 19:15:55.852336 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.852322 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 19:15:55.854297 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.854285 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 19:15:55.854330 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.854316 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 19:15:55.854490 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.854480 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 19:15:55.854521 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.854496 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 19:15:55.854521 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.854503 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 19:15:55.854521 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.854513 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 19:15:55.854521 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.854521 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 19:15:55.854616 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.854528 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 19:15:55.855089 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.855080 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 19:15:55.855117 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.855090 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 19:15:55.855117 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.855097 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 19:15:55.855117 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.855104 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 19:15:55.855117 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.855117 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 19:15:55.855291 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.855127 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 19:15:55.856441 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.856430 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 19:15:55.856478 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.856445 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 19:15:55.860486 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.860466 2578 server.go:1263] "Started kubelet" Oct 31 19:15:55.860647 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.860579 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 19:15:55.860726 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.860712 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 19:15:55.860757 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.860623 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 19:15:55.861367 ip-10-0-128-164 systemd[1]: Started Kubernetes Kubelet. Oct 31 19:15:55.863225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.863106 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 19:15:55.864402 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.864390 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 31 19:15:55.866313 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:55.866292 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 19:15:55.867932 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.867909 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 19:15:55.869359 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.869344 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 19:15:55.869525 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:55.869502 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:55.870155 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.870133 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 19:15:55.870243 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.870160 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 19:15:55.870243 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.870132 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 19:15:55.870326 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.870261 2578 factory.go:55] Registering systemd factory Oct 31 19:15:55.870326 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.870275 2578 factory.go:223] Registration of the systemd container factory successfully Oct 31 19:15:55.870326 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.870262 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 31 19:15:55.870326 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.870318 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 31 19:15:55.870494 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.870471 2578 factory.go:153] Registering CRI-O factory Oct 31 19:15:55.870494 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.870482 2578 factory.go:223] Registration of the crio container factory successfully Oct 31 19:15:55.870569 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.870540 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 19:15:55.870601 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.870570 2578 factory.go:103] Registering Raw factory Oct 31 19:15:55.870601 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.870586 2578 manager.go:1199] Started watching for new ooms in manager Oct 31 19:15:55.871658 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.871645 2578 manager.go:319] Starting recovery of all containers Oct 31 19:15:55.882395 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.882371 2578 manager.go:324] Recovery completed Oct 31 19:15:55.886912 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.886895 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:55.888856 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.888832 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:55.888949 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.888874 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:55.888949 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.888904 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:55.889444 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.889428 2578 cpu_manager.go:226] "Starting" policy="none" Oct 31 19:15:55.889444 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.889441 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 19:15:55.889540 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.889456 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 19:15:55.889540 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.889484 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:55.890360 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.890339 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:55.890439 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.890371 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:55.890439 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.890383 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:55.890703 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.890691 2578 policy_none.go:50] "Start" Oct 31 19:15:55.890760 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.890710 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 19:15:55.890760 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.890723 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 19:15:55.892119 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.892106 2578 policy_none.go:44] "Start" Oct 31 19:15:55.921510 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.921479 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 19:15:55.925522 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.925503 2578 manager.go:341] "Starting Device Plugin manager" Oct 31 19:15:55.926062 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:55.926043 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 19:15:55.926062 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.926062 2578 server.go:87] "Starting device plugin registration server" Oct 31 19:15:55.926341 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.926329 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 19:15:55.926390 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.926345 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 19:15:55.926461 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.926446 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 19:15:55.926562 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.926550 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 19:15:55.926562 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.926562 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 19:15:55.927390 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:55.927371 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 19:15:55.927472 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:55.927419 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:55.988916 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:55.988865 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-164.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 19:15:55.988916 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:55.988892 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-164.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 19:15:55.989989 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:55.988873 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-164.us-west-2.compute.internal.18e3b33e15e7b7cb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-164.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-164.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 19:15:55.860441035 +0000 UTC m=+0.409111174,LastTimestamp:2026-10-31 19:15:55.860441035 +0000 UTC m=+0.409111174,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-164.us-west-2.compute.internal,}" Oct 31 19:15:56.027416 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.027346 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.028381 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.028355 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.028510 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.028388 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.028510 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.028403 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.028510 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.028430 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.053198 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.043030 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-164.us-west-2.compute.internal.18e3b33e17994cce default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-164.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-164.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-164.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 19:15:55.88885627 +0000 UTC m=+0.437526413,LastTimestamp:2026-10-31 19:15:55.88885627 +0000 UTC m=+0.437526413,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-164.us-west-2.compute.internal,}" Oct 31 19:15:56.053198 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.043097 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 19:15:56.053198 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.043150 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 19:15:56.053198 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.043224 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-164.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 19:15:56.081410 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.081382 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 19:15:56.081505 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.081462 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cd5hl" Oct 31 19:15:56.085134 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.085106 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 19:15:56.086329 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.086311 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 19:15:56.086386 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.086338 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 19:15:56.086386 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.086360 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 19:15:56.086462 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.086398 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 19:15:56.137178 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.137005 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:15:56.142894 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.142864 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.143001 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.142906 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-164.us-west-2.compute.internal\": node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:56.186994 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.186969 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal"] Oct 31 19:15:56.187054 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.187042 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.188423 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.188408 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.188513 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.188435 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.188513 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.188445 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.190539 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.190528 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.190684 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.190669 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.190726 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.190703 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.191296 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.191279 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.191388 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.191306 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.191388 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.191316 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.191388 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.191285 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.191388 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.191371 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.191388 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.191380 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.193265 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.193250 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.193351 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.193272 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.194042 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.194026 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.194151 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.194053 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.194151 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.194066 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.226725 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.226699 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" node="ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.228637 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.228617 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" node="ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.252837 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.252818 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:56.272127 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.272085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7cd4cda822aa913fd3b1dccd57c5cf54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"7cd4cda822aa913fd3b1dccd57c5cf54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.272189 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.272133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cd4cda822aa913fd3b1dccd57c5cf54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"7cd4cda822aa913fd3b1dccd57c5cf54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.272189 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.272159 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d2e8e59ab02a8ff64cf5b536f467713-config\") pod \"kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"3d2e8e59ab02a8ff64cf5b536f467713\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.353990 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.353916 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:56.372303 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.372279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7cd4cda822aa913fd3b1dccd57c5cf54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"7cd4cda822aa913fd3b1dccd57c5cf54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.372398 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.372311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cd4cda822aa913fd3b1dccd57c5cf54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"7cd4cda822aa913fd3b1dccd57c5cf54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.372398 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.372332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d2e8e59ab02a8ff64cf5b536f467713-config\") pod \"kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"3d2e8e59ab02a8ff64cf5b536f467713\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.372398 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.372392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7cd4cda822aa913fd3b1dccd57c5cf54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"7cd4cda822aa913fd3b1dccd57c5cf54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.372503 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.372395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d2e8e59ab02a8ff64cf5b536f467713-config\") pod \"kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"3d2e8e59ab02a8ff64cf5b536f467713\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.372503 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.372395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cd4cda822aa913fd3b1dccd57c5cf54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"7cd4cda822aa913fd3b1dccd57c5cf54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.454008 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.453979 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:56.530722 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.530693 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.532825 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.532805 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:56.554490 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.554466 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:56.655000 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.654916 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:56.755457 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.755412 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:56.799778 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.799746 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 19:15:56.800265 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:56.799982 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 19:15:56.800265 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.800009 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa8143aa486024233b53a2014e030d47-5fe5f50e5295cc6b.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-164.us-west-2.compute.internal.18e3b33e179a1dc4\": read tcp 10.0.128.164:55920->18.224.159.237:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-164.us-west-2.compute.internal.18e3b33e179a1dc4 default 5183 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-164.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-164.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-164.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 19:15:55 +0000 UTC,LastTimestamp:2026-10-31 19:15:56.188449324 +0000 UTC m=+0.737119465,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-164.us-west-2.compute.internal,}" Oct 31 19:15:56.855726 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.855681 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:56.957156 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:56.957058 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:57.018609 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.018582 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:57.019605 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.019587 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:57.019717 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.019624 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:57.019717 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.019639 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:57.057713 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:57.057674 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:57.088014 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.087978 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 19:10:56 +0000 UTC" deadline="2028-06-28 23:36:19.850351852 +0000 UTC" Oct 31 19:15:57.088014 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.088011 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14548h20m22.762345721s" Oct 31 19:15:57.158720 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:57.158691 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:57.161660 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.161645 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:15:57.259604 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:57.259553 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:57.360110 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:57.360075 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:15:57.385846 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.385823 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:15:57.470095 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.470068 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:57.485897 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.485876 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:15:57.531477 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.531423 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:15:57.532534 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.532521 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:15:57.591190 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.591167 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:15:57.727394 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:57.727364 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2e8e59ab02a8ff64cf5b536f467713.slice/crio-d64471645d3a8476fc189257c351d44f1801c4de7558ebe4f9b8c740bbd3617e WatchSource:0}: Error finding container d64471645d3a8476fc189257c351d44f1801c4de7558ebe4f9b8c740bbd3617e: Status 404 returned error can't find the container with id d64471645d3a8476fc189257c351d44f1801c4de7558ebe4f9b8c740bbd3617e Oct 31 19:15:57.727924 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:57.727902 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd4cda822aa913fd3b1dccd57c5cf54.slice/crio-b529eed16eeea08a460c5cbc2490192432094395ebf6824840bc6cf8fbd64c02 WatchSource:0}: Error finding container b529eed16eeea08a460c5cbc2490192432094395ebf6824840bc6cf8fbd64c02: Status 404 returned error can't find the container with id b529eed16eeea08a460c5cbc2490192432094395ebf6824840bc6cf8fbd64c02 Oct 31 19:15:57.733313 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.733296 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:15:57.846557 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.846484 2578 apiserver.go:52] "Watching apiserver" Oct 31 19:15:57.870480 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.870448 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 19:15:57.951983 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.951948 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 19:15:57.952287 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.952264 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-xxslf","openshift-multus/multus-tdfnn","openshift-multus/network-metrics-daemon-f72jl","openshift-network-operator/iptables-alerter-k8lkj","openshift-ovn-kubernetes/ovnkube-node-s42ch","kube-system/global-pull-secret-syncer-9hfm2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl","openshift-cluster-node-tuning-operator/tuned-xk848","openshift-image-registry/node-ca-dzg4p","openshift-dns/node-resolver-h89v4","kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-7lnww","kube-system/konnectivity-agent-szn4c"] Oct 31 19:15:57.956205 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.956183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:57.958379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.958353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:15:57.958470 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:57.958436 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:15:57.960613 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.960596 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:15:57.960696 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:57.960653 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:15:57.962979 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.962944 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:15:57.965476 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.965458 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.967800 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.967783 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:15:57.967871 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:57.967840 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:15:57.970304 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.970290 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:15:57.972870 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.972852 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:57.975827 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.975807 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tdfnn" Oct 31 19:15:57.978772 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.978751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h89v4" Oct 31 19:15:57.981195 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.981177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:15:57.982664 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-systemd\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.982749 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982670 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f947add9-2413-4c26-860a-a5431e7cd2b9-dbus\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:15:57.982749 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:15:57.982749 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-socket-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:57.982749 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-kubelet\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.982749 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982733 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-cni-netd\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.982952 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-device-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:57.982952 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982785 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-system-cni-dir\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:57.982952 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982823 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:57.982952 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62dqg\" (UniqueName: \"kubernetes.io/projected/890a45ca-fe6f-4c91-9fda-034924575914-kube-api-access-62dqg\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:57.982952 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-node-log\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.982952 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982889 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovnkube-config\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.982952 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovn-node-metrics-cert\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.982952 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.982938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:57.983237 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-cnibin\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:57.983237 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-etc-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.983237 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.983237 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983103 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-cni-bin\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.983237 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-etc-selinux\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:57.983237 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:57.983237 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jz44\" (UniqueName: \"kubernetes.io/projected/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-kube-api-access-6jz44\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:15:57.983237 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983227 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-systemd-units\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983249 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-slash\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983265 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-var-lib-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-ovn\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-serviceca\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-host-slash\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-env-overrides\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovnkube-script-lib\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983386 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-registration-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-iptables-alerter-script\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-run-netns\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-log-socket\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpck4\" (UniqueName: \"kubernetes.io/projected/8d3cf005-2850-4952-b7ed-7aeeb299603e-kube-api-access-bpck4\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983526 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-host\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlppw\" (UniqueName: \"kubernetes.io/projected/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-kube-api-access-zlppw\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:15:57.983571 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983577 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:57.984211 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:57.984211 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.984211 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983641 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.984211 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2d7k\" (UniqueName: \"kubernetes.io/projected/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-kube-api-access-x2d7k\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:57.984211 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983704 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f947add9-2413-4c26-860a-a5431e7cd2b9-kubelet-config\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:15:57.984211 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983743 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-sys-fs\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:57.984211 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcwm6\" (UniqueName: \"kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6\") pod \"network-check-target-7lnww\" (UID: \"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe\") " pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:15:57.984211 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983797 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:15:57.984211 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:57.983837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw4wt\" (UniqueName: \"kubernetes.io/projected/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-kube-api-access-pw4wt\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:15:58.010368 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.010345 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 19:15:58.010473 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.010444 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vgfpk\"" Oct 31 19:15:58.010674 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.010653 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.010759 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.010656 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.058248 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.058223 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 19:15:58.058345 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.058260 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8vqt8" Oct 31 19:15:58.071793 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.071772 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 19:15:58.084103 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-socket-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.084204 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-kubelet\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084204 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-cni-netd\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084204 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-modprobe-d\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.084204 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-systemd\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.084384 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084202 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-kubelet\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084384 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-host\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.084384 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084235 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-cnibin\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.084384 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084238 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-cni-netd\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084384 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-socket-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.084384 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.084384 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovnkube-config\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084384 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovn-node-metrics-cert\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084384 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-k8s-cni-cncf-io\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.084384 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-cni-multus\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-daemon-config\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-etc-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-cni-bin\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-cni-bin\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084481 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-etc-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084505 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0875c4d7-7bfd-45ab-ae34-75e20e327605-hosts-file\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysctl-conf\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084596 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-systemd-units\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084630 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-systemd-units\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084661 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-ovn\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-env-overrides\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084713 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0875c4d7-7bfd-45ab-ae34-75e20e327605-tmp-dir\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084740 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-ovn\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-kubernetes\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-host-slash\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:15:58.084848 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/474d47d1-f185-4127-a085-786c6ada462e-agent-certs\") pod \"konnectivity-agent-szn4c\" (UID: \"474d47d1-f185-4127-a085-786c6ada462e\") " pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysconfig\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-multus-certs\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084869 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-host-slash\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-log-socket\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084942 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-log-socket\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysctl-d\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-kubelet\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.084991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-conf-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpck4\" (UniqueName: \"kubernetes.io/projected/8d3cf005-2850-4952-b7ed-7aeeb299603e-kube-api-access-bpck4\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlppw\" (UniqueName: \"kubernetes.io/projected/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-kube-api-access-zlppw\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-var-lib-kubelet\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085282 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-tuned\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.085484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d59s2\" (UniqueName: \"kubernetes.io/projected/82d520a7-fab4-46a7-bed2-9d4b3291d198-kube-api-access-d59s2\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085315 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-sys-fs\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.085367 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085378 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-sys-fs\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.085457 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret podName:f947add9-2413-4c26-860a-a5431e7cd2b9 nodeName:}" failed. No retries permitted until 2026-10-31 19:15:58.585439805 +0000 UTC m=+3.134109946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret") pod "global-pull-secret-syncer-9hfm2" (UID: "f947add9-2413-4c26-860a-a5431e7cd2b9") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-socket-dir-parent\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.085529 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9k9l\" (UniqueName: \"kubernetes.io/projected/3de0cf07-8e8a-47c6-a382-bc56884867e6-kube-api-access-f9k9l\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-lib-modules\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-device-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.085610 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs podName:d27497f4-b29e-46c2-8f63-664fbe7e8dd1 nodeName:}" failed. No retries permitted until 2026-10-31 19:15:58.585601925 +0000 UTC m=+3.134272056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs") pod "network-metrics-daemon-f72jl" (UID: "d27497f4-b29e-46c2-8f63-664fbe7e8dd1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-system-cni-dir\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-device-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-system-cni-dir\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.086225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62dqg\" (UniqueName: \"kubernetes.io/projected/890a45ca-fe6f-4c91-9fda-034924575914-kube-api-access-62dqg\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-node-log\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-cni-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085724 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3de0cf07-8e8a-47c6-a382-bc56884867e6-cni-binary-copy\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085745 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-node-log\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-cnibin\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085784 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-cnibin\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085813 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngplm\" (UniqueName: \"kubernetes.io/projected/0875c4d7-7bfd-45ab-ae34-75e20e327605-kube-api-access-ngplm\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-etc-kubernetes\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-etc-selinux\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz44\" (UniqueName: \"kubernetes.io/projected/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-kube-api-access-6jz44\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.085982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-etc-selinux\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.086899 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-slash\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-var-lib-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086051 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-run\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086070 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-var-lib-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086068 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-slash\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82d520a7-fab4-46a7-bed2-9d4b3291d198-tmp\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-serviceca\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovnkube-script-lib\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-netns\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-registration-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-iptables-alerter-script\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-run-netns\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-sys\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086283 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-registration-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-hostroot\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086343 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-run-netns\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.087379 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-host\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086430 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2d7k\" (UniqueName: \"kubernetes.io/projected/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-kube-api-access-x2d7k\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086440 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-host\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086454 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f947add9-2413-4c26-860a-a5431e7cd2b9-kubelet-config\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f947add9-2413-4c26-860a-a5431e7cd2b9-kubelet-config\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/474d47d1-f185-4127-a085-786c6ada462e-konnectivity-ca\") pod \"konnectivity-agent-szn4c\" (UID: \"474d47d1-f185-4127-a085-786c6ada462e\") " pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-cni-bin\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwm6\" (UniqueName: \"kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6\") pod \"network-check-target-7lnww\" (UID: \"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe\") " pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4wt\" (UniqueName: \"kubernetes.io/projected/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-kube-api-access-pw4wt\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-systemd\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086687 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-systemd\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f947add9-2413-4c26-860a-a5431e7cd2b9-dbus\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086730 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-system-cni-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.087883 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.086936 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f947add9-2413-4c26-860a-a5431e7cd2b9-dbus\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:15:58.091087 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.091048 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" event={"ID":"7cd4cda822aa913fd3b1dccd57c5cf54","Type":"ContainerStarted","Data":"b529eed16eeea08a460c5cbc2490192432094395ebf6824840bc6cf8fbd64c02"} Oct 31 19:15:58.092117 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.092075 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" event={"ID":"3d2e8e59ab02a8ff64cf5b536f467713","Type":"ContainerStarted","Data":"d64471645d3a8476fc189257c351d44f1801c4de7558ebe4f9b8c740bbd3617e"} Oct 31 19:15:58.119457 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119378 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 19:15:58.119457 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119378 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.119457 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119378 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 19:15:58.119457 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119414 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 19:15:58.119457 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119441 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.119729 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119475 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.119797 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119778 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.119854 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119843 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.119910 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119855 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z8492\"" Oct 31 19:15:58.119910 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119878 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tqdrb\"" Oct 31 19:15:58.119910 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119908 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 19:15:58.120082 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119927 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.120082 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119935 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.120082 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119945 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.120082 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119939 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 19:15:58.120082 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.119968 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 19:15:58.120082 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.120002 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 19:15:58.120082 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.120012 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.120300 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.120160 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2kswh\"" Oct 31 19:15:58.120300 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.120204 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 19:15:58.124974 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.124934 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovnkube-config\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.124974 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.124947 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.125245 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.125225 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-env-overrides\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.125381 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.125362 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.127234 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.127213 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-serviceca\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:15:58.127316 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.127239 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.127388 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.127374 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.127510 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.127495 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.127698 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.127685 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 19:15:58.127750 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.127737 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.127799 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.127755 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 19:15:58.127799 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.127786 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hlrbg\"" Oct 31 19:15:58.135358 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.135338 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 19:15:58.137156 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.137137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-iptables-alerter-script\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:15:58.138814 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.138797 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovn-node-metrics-cert\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.171153 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.171127 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lwj7k\"" Oct 31 19:15:58.173160 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.171148 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zsjk9\"" Oct 31 19:15:58.173160 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.171668 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4ftk8\"" Oct 31 19:15:58.173160 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.171694 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 19:15:58.173160 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.171757 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d7wnz\"" Oct 31 19:15:58.173160 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.171862 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 19:15:58.173160 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.173591 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:15:58.173160 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.173626 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:15:58.173160 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.173639 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vcwm6 for pod openshift-network-diagnostics/network-check-target-7lnww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:15:58.173160 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.173740 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6 podName:64d8b12c-0fc1-4b11-9a11-8155f18c8dbe nodeName:}" failed. No retries permitted until 2026-10-31 19:15:58.673718883 +0000 UTC m=+3.222389042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vcwm6" (UniqueName: "kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6") pod "network-check-target-7lnww" (UID: "64d8b12c-0fc1-4b11-9a11-8155f18c8dbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:15:58.175297 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.175156 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlppw\" (UniqueName: \"kubernetes.io/projected/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-kube-api-access-zlppw\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:15:58.175752 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.175702 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62dqg\" (UniqueName: \"kubernetes.io/projected/890a45ca-fe6f-4c91-9fda-034924575914-kube-api-access-62dqg\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.176908 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.176830 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2d7k\" (UniqueName: \"kubernetes.io/projected/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-kube-api-access-x2d7k\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.176908 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.176879 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jz44\" (UniqueName: \"kubernetes.io/projected/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-kube-api-access-6jz44\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:15:58.177106 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.176909 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovnkube-script-lib\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.177106 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.176916 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpck4\" (UniqueName: \"kubernetes.io/projected/8d3cf005-2850-4952-b7ed-7aeeb299603e-kube-api-access-bpck4\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.177201 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.177182 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4wt\" (UniqueName: \"kubernetes.io/projected/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-kube-api-access-pw4wt\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:15:58.187303 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngplm\" (UniqueName: \"kubernetes.io/projected/0875c4d7-7bfd-45ab-ae34-75e20e327605-kube-api-access-ngplm\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:15:58.187355 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-etc-kubernetes\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187355 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-run\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.187355 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82d520a7-fab4-46a7-bed2-9d4b3291d198-tmp\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.187507 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-netns\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187507 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-sys\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.187507 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187394 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-etc-kubernetes\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187507 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-hostroot\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187507 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187427 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-run\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.187507 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/474d47d1-f185-4127-a085-786c6ada462e-konnectivity-ca\") pod \"konnectivity-agent-szn4c\" (UID: \"474d47d1-f185-4127-a085-786c6ada462e\") " pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:15:58.187507 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187448 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-netns\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187507 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-cni-bin\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-sys\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-system-cni-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187562 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-cni-bin\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187513 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-hostroot\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-modprobe-d\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-systemd\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-system-cni-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-host\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187673 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-cnibin\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187683 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-host\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187679 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-systemd\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-k8s-cni-cncf-io\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187724 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-cnibin\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-cni-multus\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187740 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-modprobe-d\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-daemon-config\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187773 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-k8s-cni-cncf-io\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.187878 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187781 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-cni-multus\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0875c4d7-7bfd-45ab-ae34-75e20e327605-hosts-file\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysctl-conf\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0875c4d7-7bfd-45ab-ae34-75e20e327605-tmp-dir\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-kubernetes\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187843 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0875c4d7-7bfd-45ab-ae34-75e20e327605-hosts-file\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-kubernetes\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/474d47d1-f185-4127-a085-786c6ada462e-agent-certs\") pod \"konnectivity-agent-szn4c\" (UID: \"474d47d1-f185-4127-a085-786c6ada462e\") " pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysconfig\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysctl-conf\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.187988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-multus-certs\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188025 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-multus-certs\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysctl-d\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-kubelet\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-conf-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188100 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/474d47d1-f185-4127-a085-786c6ada462e-konnectivity-ca\") pod \"konnectivity-agent-szn4c\" (UID: \"474d47d1-f185-4127-a085-786c6ada462e\") " pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysctl-d\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-var-lib-kubelet\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.188667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188144 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-conf-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-tuned\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188186 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysconfig\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188187 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0875c4d7-7bfd-45ab-ae34-75e20e327605-tmp-dir\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188203 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-var-lib-kubelet\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188206 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d59s2\" (UniqueName: \"kubernetes.io/projected/82d520a7-fab4-46a7-bed2-9d4b3291d198-kube-api-access-d59s2\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188226 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-kubelet\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-socket-dir-parent\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9k9l\" (UniqueName: \"kubernetes.io/projected/3de0cf07-8e8a-47c6-a382-bc56884867e6-kube-api-access-f9k9l\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-lib-modules\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188491 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-lib-modules\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188500 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-socket-dir-parent\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-cni-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3de0cf07-8e8a-47c6-a382-bc56884867e6-cni-binary-copy\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.188586 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-cni-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.189008 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-daemon-config\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.189424 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.189221 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3de0cf07-8e8a-47c6-a382-bc56884867e6-cni-binary-copy\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.189890 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.189696 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82d520a7-fab4-46a7-bed2-9d4b3291d198-tmp\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.190161 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.190143 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/474d47d1-f185-4127-a085-786c6ada462e-agent-certs\") pod \"konnectivity-agent-szn4c\" (UID: \"474d47d1-f185-4127-a085-786c6ada462e\") " pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:15:58.190583 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.190566 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-tuned\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.245455 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.245424 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngplm\" (UniqueName: \"kubernetes.io/projected/0875c4d7-7bfd-45ab-ae34-75e20e327605-kube-api-access-ngplm\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:15:58.245833 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.245811 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d59s2\" (UniqueName: \"kubernetes.io/projected/82d520a7-fab4-46a7-bed2-9d4b3291d198-kube-api-access-d59s2\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.245893 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.245831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9k9l\" (UniqueName: \"kubernetes.io/projected/3de0cf07-8e8a-47c6-a382-bc56884867e6-kube-api-access-f9k9l\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.272267 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.272246 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:15:58.277493 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.277473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:15:58.283051 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:58.283024 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3cf005_2850_4952_b7ed_7aeeb299603e.slice/crio-fa92ed98c3bfc125b180abc5645a89a148875f624a3dfcfc2c88060eef095df2 WatchSource:0}: Error finding container fa92ed98c3bfc125b180abc5645a89a148875f624a3dfcfc2c88060eef095df2: Status 404 returned error can't find the container with id fa92ed98c3bfc125b180abc5645a89a148875f624a3dfcfc2c88060eef095df2 Oct 31 19:15:58.287348 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:58.287323 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b056a7_e321_4ce7_ab1e_0d2e2824869f.slice/crio-6056b78227e51f31f0bb4e7b7c0cccb905810522bb003cec3fd9753ad5a9cf35 WatchSource:0}: Error finding container 6056b78227e51f31f0bb4e7b7c0cccb905810522bb003cec3fd9753ad5a9cf35: Status 404 returned error can't find the container with id 6056b78227e51f31f0bb4e7b7c0cccb905810522bb003cec3fd9753ad5a9cf35 Oct 31 19:15:58.297669 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.297649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:15:58.306681 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:58.306657 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be4a9b0_a460_4e50_aa8a_961f5a3e4d7f.slice/crio-223beb27e8090e99901d90d5ea58cae2b9a5cf3f3de4eb899f4524ca9c8c2c02 WatchSource:0}: Error finding container 223beb27e8090e99901d90d5ea58cae2b9a5cf3f3de4eb899f4524ca9c8c2c02: Status 404 returned error can't find the container with id 223beb27e8090e99901d90d5ea58cae2b9a5cf3f3de4eb899f4524ca9c8c2c02 Oct 31 19:15:58.307255 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.307236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tdfnn" Oct 31 19:15:58.310057 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.310040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:15:58.314926 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.314907 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h89v4" Oct 31 19:15:58.315108 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.315093 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:15:58.316784 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.316765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:15:58.317012 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:58.316988 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de0cf07_8e8a_47c6_a382_bc56884867e6.slice/crio-95d91696c8a2a094ee217a9ec78b15c42fc68cdd354e4db542ed2ed08c638006 WatchSource:0}: Error finding container 95d91696c8a2a094ee217a9ec78b15c42fc68cdd354e4db542ed2ed08c638006: Status 404 returned error can't find the container with id 95d91696c8a2a094ee217a9ec78b15c42fc68cdd354e4db542ed2ed08c638006 Oct 31 19:15:58.320008 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:58.319982 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c9d77e_0f26_4a74_a08b_a2aff00389d2.slice/crio-6f4eb791676db0e46c79c668c61b8f65927d25d6a942bd9d7a49d5657b7e34c9 WatchSource:0}: Error finding container 6f4eb791676db0e46c79c668c61b8f65927d25d6a942bd9d7a49d5657b7e34c9: Status 404 returned error can't find the container with id 6f4eb791676db0e46c79c668c61b8f65927d25d6a942bd9d7a49d5657b7e34c9 Oct 31 19:15:58.326156 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.326133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:15:58.327751 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.327732 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:15:58.328003 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:58.327985 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0875c4d7_7bfd_45ab_ae34_75e20e327605.slice/crio-82fbb210bc636df8d5b25f0f20c6ea4598aecfe78a54daf7227152438d0386ce WatchSource:0}: Error finding container 82fbb210bc636df8d5b25f0f20c6ea4598aecfe78a54daf7227152438d0386ce: Status 404 returned error can't find the container with id 82fbb210bc636df8d5b25f0f20c6ea4598aecfe78a54daf7227152438d0386ce Oct 31 19:15:58.328905 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:58.328874 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474d47d1_f185_4127_a085_786c6ada462e.slice/crio-5124428798b63af291653d7e0cd2cb62ef3da0b924cdd015b46d27e445abf261 WatchSource:0}: Error finding container 5124428798b63af291653d7e0cd2cb62ef3da0b924cdd015b46d27e445abf261: Status 404 returned error can't find the container with id 5124428798b63af291653d7e0cd2cb62ef3da0b924cdd015b46d27e445abf261 Oct 31 19:15:58.335477 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:58.335455 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d520a7_fab4_46a7_bed2_9d4b3291d198.slice/crio-7ce741db11b471b17b14942b1ffea609c3bbb62ddd69115d95d9b8dddf211869 WatchSource:0}: Error finding container 7ce741db11b471b17b14942b1ffea609c3bbb62ddd69115d95d9b8dddf211869: Status 404 returned error can't find the container with id 7ce741db11b471b17b14942b1ffea609c3bbb62ddd69115d95d9b8dddf211869 Oct 31 19:15:58.338641 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:15:58.338621 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890a45ca_fe6f_4c91_9fda_034924575914.slice/crio-5735a738be1b3e5df12b00db7aaf9420ec75770bcb20600c3fde66fa606c165f WatchSource:0}: Error finding container 5735a738be1b3e5df12b00db7aaf9420ec75770bcb20600c3fde66fa606c165f: Status 404 returned error can't find the container with id 5735a738be1b3e5df12b00db7aaf9420ec75770bcb20600c3fde66fa606c165f Oct 31 19:15:58.591266 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.591172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:15:58.591266 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.591225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:15:58.591471 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.591344 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:15:58.591471 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.591413 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs podName:d27497f4-b29e-46c2-8f63-664fbe7e8dd1 nodeName:}" failed. No retries permitted until 2026-10-31 19:15:59.591391455 +0000 UTC m=+4.140061610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs") pod "network-metrics-daemon-f72jl" (UID: "d27497f4-b29e-46c2-8f63-664fbe7e8dd1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:15:58.591471 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.591342 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:15:58.591471 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.591468 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret podName:f947add9-2413-4c26-860a-a5431e7cd2b9 nodeName:}" failed. No retries permitted until 2026-10-31 19:15:59.591456435 +0000 UTC m=+4.140126567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret") pod "global-pull-secret-syncer-9hfm2" (UID: "f947add9-2413-4c26-860a-a5431e7cd2b9") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:15:58.691679 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:58.691646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwm6\" (UniqueName: \"kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6\") pod \"network-check-target-7lnww\" (UID: \"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe\") " pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:15:58.691854 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.691781 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:15:58.691854 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.691794 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:15:58.691854 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.691802 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vcwm6 for pod openshift-network-diagnostics/network-check-target-7lnww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:15:58.691854 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:58.691847 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6 podName:64d8b12c-0fc1-4b11-9a11-8155f18c8dbe nodeName:}" failed. No retries permitted until 2026-10-31 19:15:59.69183371 +0000 UTC m=+4.240503838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vcwm6" (UniqueName: "kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6") pod "network-check-target-7lnww" (UID: "64d8b12c-0fc1-4b11-9a11-8155f18c8dbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:15:59.061682 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.061556 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:10:58 +0000 UTC" deadline="2028-05-07 18:07:24.046670257 +0000 UTC" Oct 31 19:15:59.061682 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.061620 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13294h51m24.985055625s" Oct 31 19:15:59.095698 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.095655 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xk848" event={"ID":"82d520a7-fab4-46a7-bed2-9d4b3291d198","Type":"ContainerStarted","Data":"7ce741db11b471b17b14942b1ffea609c3bbb62ddd69115d95d9b8dddf211869"} Oct 31 19:15:59.097349 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.097314 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" event={"ID":"f6c9d77e-0f26-4a74-a08b-a2aff00389d2","Type":"ContainerStarted","Data":"6f4eb791676db0e46c79c668c61b8f65927d25d6a942bd9d7a49d5657b7e34c9"} Oct 31 19:15:59.098628 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.098604 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tdfnn" event={"ID":"3de0cf07-8e8a-47c6-a382-bc56884867e6","Type":"ContainerStarted","Data":"95d91696c8a2a094ee217a9ec78b15c42fc68cdd354e4db542ed2ed08c638006"} Oct 31 19:15:59.099943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.099912 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dzg4p" event={"ID":"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f","Type":"ContainerStarted","Data":"223beb27e8090e99901d90d5ea58cae2b9a5cf3f3de4eb899f4524ca9c8c2c02"} Oct 31 19:15:59.101227 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.101193 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k8lkj" event={"ID":"c8b056a7-e321-4ce7-ab1e-0d2e2824869f","Type":"ContainerStarted","Data":"6056b78227e51f31f0bb4e7b7c0cccb905810522bb003cec3fd9753ad5a9cf35"} Oct 31 19:15:59.102618 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.102593 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" event={"ID":"8d3cf005-2850-4952-b7ed-7aeeb299603e","Type":"ContainerStarted","Data":"fa92ed98c3bfc125b180abc5645a89a148875f624a3dfcfc2c88060eef095df2"} Oct 31 19:15:59.104487 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.104461 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxslf" event={"ID":"890a45ca-fe6f-4c91-9fda-034924575914","Type":"ContainerStarted","Data":"5735a738be1b3e5df12b00db7aaf9420ec75770bcb20600c3fde66fa606c165f"} Oct 31 19:15:59.105978 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.105871 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-szn4c" event={"ID":"474d47d1-f185-4127-a085-786c6ada462e","Type":"ContainerStarted","Data":"5124428798b63af291653d7e0cd2cb62ef3da0b924cdd015b46d27e445abf261"} Oct 31 19:15:59.107400 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.107378 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h89v4" event={"ID":"0875c4d7-7bfd-45ab-ae34-75e20e327605","Type":"ContainerStarted","Data":"82fbb210bc636df8d5b25f0f20c6ea4598aecfe78a54daf7227152438d0386ce"} Oct 31 19:15:59.600669 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.600620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:15:59.600863 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.600687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:15:59.601014 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:59.600993 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:15:59.601086 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:59.601073 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs podName:d27497f4-b29e-46c2-8f63-664fbe7e8dd1 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:01.601051574 +0000 UTC m=+6.149721723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs") pod "network-metrics-daemon-f72jl" (UID: "d27497f4-b29e-46c2-8f63-664fbe7e8dd1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:15:59.601180 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:59.601166 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:15:59.601233 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:59.601213 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret podName:f947add9-2413-4c26-860a-a5431e7cd2b9 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:01.601200947 +0000 UTC m=+6.149871091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret") pod "global-pull-secret-syncer-9hfm2" (UID: "f947add9-2413-4c26-860a-a5431e7cd2b9") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:15:59.702202 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:15:59.702149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwm6\" (UniqueName: \"kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6\") pod \"network-check-target-7lnww\" (UID: \"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe\") " pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:15:59.702434 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:59.702415 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:15:59.702509 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:59.702443 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:15:59.702509 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:59.702458 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vcwm6 for pod openshift-network-diagnostics/network-check-target-7lnww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:15:59.702611 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:15:59.702540 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6 podName:64d8b12c-0fc1-4b11-9a11-8155f18c8dbe nodeName:}" failed. No retries permitted until 2026-10-31 19:16:01.702518846 +0000 UTC m=+6.251188989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vcwm6" (UniqueName: "kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6") pod "network-check-target-7lnww" (UID: "64d8b12c-0fc1-4b11-9a11-8155f18c8dbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:00.062232 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:00.062116 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:10:58 +0000 UTC" deadline="2028-06-21 01:01:24.96174005 +0000 UTC" Oct 31 19:16:00.062232 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:00.062179 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14357h45m24.899564495s" Oct 31 19:16:00.089892 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:00.087151 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:00.089892 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:00.087294 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:00.089892 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:00.087400 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:00.089892 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:00.087486 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:00.089892 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:00.087545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:00.089892 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:00.087604 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:01.620083 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:01.619992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:01.620083 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:01.620055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:01.620635 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:01.620244 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:01.620635 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:01.620313 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs podName:d27497f4-b29e-46c2-8f63-664fbe7e8dd1 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:05.620292909 +0000 UTC m=+10.168963040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs") pod "network-metrics-daemon-f72jl" (UID: "d27497f4-b29e-46c2-8f63-664fbe7e8dd1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:01.620635 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:01.620381 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:01.620635 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:01.620417 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret podName:f947add9-2413-4c26-860a-a5431e7cd2b9 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:05.620405095 +0000 UTC m=+10.169075223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret") pod "global-pull-secret-syncer-9hfm2" (UID: "f947add9-2413-4c26-860a-a5431e7cd2b9") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:01.721771 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:01.721163 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwm6\" (UniqueName: \"kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6\") pod \"network-check-target-7lnww\" (UID: \"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe\") " pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:01.721771 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:01.721417 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:16:01.721771 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:01.721440 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:16:01.721771 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:01.721454 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vcwm6 for pod openshift-network-diagnostics/network-check-target-7lnww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:01.721771 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:01.721521 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6 podName:64d8b12c-0fc1-4b11-9a11-8155f18c8dbe nodeName:}" failed. No retries permitted until 2026-10-31 19:16:05.721501463 +0000 UTC m=+10.270171593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vcwm6" (UniqueName: "kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6") pod "network-check-target-7lnww" (UID: "64d8b12c-0fc1-4b11-9a11-8155f18c8dbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:02.088992 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:02.087682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:02.090333 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:02.089254 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:02.090333 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:02.089329 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:02.090333 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:02.089410 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:02.090333 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:02.089450 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:02.090333 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:02.089526 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:03.136308 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:03.134967 2578 generic.go:356] "Generic (PLEG): container finished" podID="7cd4cda822aa913fd3b1dccd57c5cf54" containerID="56425514f58539f94d83e1c7b8457cd7d82d045dfdc773b5845ab0db68b76aae" exitCode=0 Oct 31 19:16:03.136308 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:03.135303 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" event={"ID":"7cd4cda822aa913fd3b1dccd57c5cf54","Type":"ContainerDied","Data":"56425514f58539f94d83e1c7b8457cd7d82d045dfdc773b5845ab0db68b76aae"} Oct 31 19:16:03.142665 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:03.142437 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" event={"ID":"3d2e8e59ab02a8ff64cf5b536f467713","Type":"ContainerStarted","Data":"ae9cf82de1dbef56205b7393cc6aa2c7ae83f4cbc100a232d24b17a91d88d315"} Oct 31 19:16:03.151087 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:03.150290 2578 generic.go:356] "Generic (PLEG): container finished" podID="890a45ca-fe6f-4c91-9fda-034924575914" containerID="72af8a74067443f6b078f0050416afaeb2cc3a8f50522c78ae557a694e865c97" exitCode=0 Oct 31 19:16:03.151087 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:03.150358 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxslf" event={"ID":"890a45ca-fe6f-4c91-9fda-034924575914","Type":"ContainerDied","Data":"72af8a74067443f6b078f0050416afaeb2cc3a8f50522c78ae557a694e865c97"} Oct 31 19:16:03.369943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:03.369878 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" podStartSLOduration=5.369861896 podStartE2EDuration="5.369861896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:03.368905571 +0000 UTC m=+7.917575723" watchObservedRunningTime="2026-10-31 19:16:03.369861896 +0000 UTC m=+7.918532047" Oct 31 19:16:04.089176 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:04.089130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:04.089348 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:04.089265 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:04.089420 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:04.089355 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:04.089491 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:04.089466 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:04.089541 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:04.089524 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:04.089614 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:04.089594 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:04.158113 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:04.158042 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-szn4c" event={"ID":"474d47d1-f185-4127-a085-786c6ada462e","Type":"ContainerStarted","Data":"2919c32d927a1e0a41aa0ff74d81f8e05e844b9bb7d77a6edd78c63b33a9669d"} Oct 31 19:16:04.163812 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:04.163772 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" event={"ID":"7cd4cda822aa913fd3b1dccd57c5cf54","Type":"ContainerStarted","Data":"d0c8e31d12262fa034e83abd08c3b9241481b5efed0a26df44d7660aeaa0cce1"} Oct 31 19:16:04.273780 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:04.273727 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-szn4c" podStartSLOduration=3.451061611 podStartE2EDuration="8.273711004s" totalImagesPullingTime="4.822649393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:04.273696624 +0000 UTC m=+8.822366776" watchObservedRunningTime="2026-10-31 19:16:04.273711004 +0000 UTC m=+8.822381136" Oct 31 19:16:05.179761 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:05.179216 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dzg4p" event={"ID":"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f","Type":"ContainerStarted","Data":"19de29b7da803705352a821c396686bf9d50e11e2151268e9be5817efd0aad79"} Oct 31 19:16:05.286649 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:05.285855 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" podStartSLOduration=7.285834547 podStartE2EDuration="7.285834547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:04.389542169 +0000 UTC m=+8.938212322" watchObservedRunningTime="2026-10-31 19:16:05.285834547 +0000 UTC m=+9.834504701" Oct 31 19:16:05.286649 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:05.286511 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dzg4p" podStartSLOduration=3.335221817 podStartE2EDuration="9.286501701s" totalImagesPullingTime="5.951279884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:05.285826505 +0000 UTC m=+9.834496658" watchObservedRunningTime="2026-10-31 19:16:05.286501701 +0000 UTC m=+9.835171849" Oct 31 19:16:05.660695 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:05.660279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:05.660695 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:05.660335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:05.660695 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:05.660488 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:05.660695 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:05.660548 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs podName:d27497f4-b29e-46c2-8f63-664fbe7e8dd1 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:13.660531396 +0000 UTC m=+18.209201527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs") pod "network-metrics-daemon-f72jl" (UID: "d27497f4-b29e-46c2-8f63-664fbe7e8dd1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:05.660695 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:05.660610 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:05.660695 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:05.660640 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret podName:f947add9-2413-4c26-860a-a5431e7cd2b9 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:13.660629967 +0000 UTC m=+18.209300096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret") pod "global-pull-secret-syncer-9hfm2" (UID: "f947add9-2413-4c26-860a-a5431e7cd2b9") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:05.761004 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:05.760650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwm6\" (UniqueName: \"kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6\") pod \"network-check-target-7lnww\" (UID: \"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe\") " pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:05.761004 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:05.760865 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:16:05.761004 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:05.760901 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:16:05.761004 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:05.760915 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vcwm6 for pod openshift-network-diagnostics/network-check-target-7lnww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:05.761004 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:05.760978 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6 podName:64d8b12c-0fc1-4b11-9a11-8155f18c8dbe nodeName:}" failed. No retries permitted until 2026-10-31 19:16:13.760962004 +0000 UTC m=+18.309632147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vcwm6" (UniqueName: "kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6") pod "network-check-target-7lnww" (UID: "64d8b12c-0fc1-4b11-9a11-8155f18c8dbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:06.088507 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:06.088443 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:06.088664 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:06.088521 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:06.088664 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:06.088595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:06.088762 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:06.088673 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:06.088950 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:06.088804 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:06.089044 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:06.088960 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:06.186783 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:06.186741 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" event={"ID":"8d3cf005-2850-4952-b7ed-7aeeb299603e","Type":"ContainerStarted","Data":"8f2f1d4b230fa13997dae74e7fb81ac2d2a3378a335fe760efd0b81ae79955cd"} Oct 31 19:16:06.431166 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:06.431093 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:16:06.432254 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:06.432232 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:16:07.191775 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:07.191699 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:16:07.193000 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:07.192790 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:16:07.405373 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:07.405276 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 19:16:07.937464 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:07.937253 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T19:16:07.405293541Z","UUID":"5dcc9fe0-b37c-42a9-95e4-1b756d4ba324","Handler":null,"Name":"","Endpoint":""} Oct 31 19:16:07.939053 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:07.939017 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 19:16:07.939053 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:07.939054 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 19:16:08.087038 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:08.086970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:08.087199 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:08.087089 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:08.087199 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:08.087163 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:08.087316 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:08.087251 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:08.087316 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:08.087312 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:08.087413 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:08.087378 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:08.195192 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:08.195155 2578 generic.go:356] "Generic (PLEG): container finished" podID="890a45ca-fe6f-4c91-9fda-034924575914" containerID="d63e43472eaa4547660bcafa605d4e3e38c2678bb7c657ed16763cc9e6ffd12f" exitCode=0 Oct 31 19:16:08.195524 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:08.195243 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxslf" event={"ID":"890a45ca-fe6f-4c91-9fda-034924575914","Type":"ContainerDied","Data":"d63e43472eaa4547660bcafa605d4e3e38c2678bb7c657ed16763cc9e6ffd12f"} Oct 31 19:16:08.198013 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:08.197758 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h89v4" event={"ID":"0875c4d7-7bfd-45ab-ae34-75e20e327605","Type":"ContainerStarted","Data":"db57afd68ec0821f959b693eb119f17f64d3342fbd0b0bccabc1616f4baaa1ef"} Oct 31 19:16:08.200207 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:08.200159 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tdfnn" event={"ID":"3de0cf07-8e8a-47c6-a382-bc56884867e6","Type":"ContainerStarted","Data":"003ebcd6b06b72794454bf6eb2faa5b6ef77f87ca7676dcd14e2a55010f6fcc7"} Oct 31 19:16:08.202869 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:08.202842 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" event={"ID":"8d3cf005-2850-4952-b7ed-7aeeb299603e","Type":"ContainerStarted","Data":"ff19a9a9ff5892f2c9f82ad2da4a28607b0ac1bdeea22e3b3e17429d980a4947"} Oct 31 19:16:08.413038 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:08.412753 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-h89v4" podStartSLOduration=3.3659995609999998 podStartE2EDuration="12.412733514s" totalImagesPullingTime="9.046733953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:08.412564006 +0000 UTC m=+12.961234153" watchObservedRunningTime="2026-10-31 19:16:08.412733514 +0000 UTC m=+12.961403663" Oct 31 19:16:08.527116 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:08.523009 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tdfnn" podStartSLOduration=3.710412733 podStartE2EDuration="12.522991369s" totalImagesPullingTime="8.812578636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:08.522983904 +0000 UTC m=+13.071654052" watchObservedRunningTime="2026-10-31 19:16:08.522991369 +0000 UTC m=+13.071661518" Oct 31 19:16:09.205856 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:09.205817 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k8lkj" event={"ID":"c8b056a7-e321-4ce7-ab1e-0d2e2824869f","Type":"ContainerStarted","Data":"ab0fa350854a1ac43d041f5e32d304c2773d6e6586f6d9db306901a63de2df49"} Oct 31 19:16:09.312729 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:09.312674 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k8lkj" podStartSLOduration=4.412642111 podStartE2EDuration="13.312662556s" totalImagesPullingTime="8.900020445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:09.312313205 +0000 UTC m=+13.860983335" watchObservedRunningTime="2026-10-31 19:16:09.312662556 +0000 UTC m=+13.861332703" Oct 31 19:16:10.087254 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:10.087219 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:10.087405 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:10.087227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:10.087405 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:10.087345 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:10.087405 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:10.087352 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:10.087536 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:10.087424 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:10.087536 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:10.087485 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:10.210183 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:10.209787 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xk848" event={"ID":"82d520a7-fab4-46a7-bed2-9d4b3291d198","Type":"ContainerStarted","Data":"960f404a6ec03b063cfbc47d831ab64b33c875d56aba04ced304214f92021886"} Oct 31 19:16:10.212953 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:10.212924 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" event={"ID":"8d3cf005-2850-4952-b7ed-7aeeb299603e","Type":"ContainerStarted","Data":"a850575ea2e164025ac5998888bfc35c0ad86839473dc7d5e394c0efac415218"} Oct 31 19:16:10.319552 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:10.319498 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xk848" podStartSLOduration=3.158758518 podStartE2EDuration="14.319482434s" totalImagesPullingTime="11.160723916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:10.319119884 +0000 UTC m=+14.867790034" watchObservedRunningTime="2026-10-31 19:16:10.319482434 +0000 UTC m=+14.868152588" Oct 31 19:16:10.435581 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:10.435540 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" podStartSLOduration=3.7830005829999998 podStartE2EDuration="14.43552613s" totalImagesPullingTime="10.652525547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:10.435085183 +0000 UTC m=+14.983755332" watchObservedRunningTime="2026-10-31 19:16:10.43552613 +0000 UTC m=+14.984196278" Oct 31 19:16:11.217371 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:11.217335 2578 generic.go:356] "Generic (PLEG): container finished" podID="890a45ca-fe6f-4c91-9fda-034924575914" containerID="2e59369e0298ba9e055dd3b38271729415abe7b7c8e76e252b082bc7e2487930" exitCode=0 Oct 31 19:16:11.218152 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:11.217441 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxslf" event={"ID":"890a45ca-fe6f-4c91-9fda-034924575914","Type":"ContainerDied","Data":"2e59369e0298ba9e055dd3b38271729415abe7b7c8e76e252b082bc7e2487930"} Oct 31 19:16:12.087332 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:12.087119 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:12.087512 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:12.087145 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:12.087512 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:12.087439 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:12.087512 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:12.087195 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:12.087669 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:12.087521 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:12.087669 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:12.087621 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:13.225532 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:13.225481 2578 generic.go:356] "Generic (PLEG): container finished" podID="890a45ca-fe6f-4c91-9fda-034924575914" containerID="04c70e41a29b4f52626db39a39925788cf1d8367e22fe8853d3b29637192ce8c" exitCode=0 Oct 31 19:16:13.225532 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:13.225547 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxslf" event={"ID":"890a45ca-fe6f-4c91-9fda-034924575914","Type":"ContainerDied","Data":"04c70e41a29b4f52626db39a39925788cf1d8367e22fe8853d3b29637192ce8c"} Oct 31 19:16:13.722641 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:13.722563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:13.722641 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:13.722603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:13.722823 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:13.722723 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:13.722823 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:13.722790 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs podName:d27497f4-b29e-46c2-8f63-664fbe7e8dd1 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:29.722778033 +0000 UTC m=+34.271448159 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs") pod "network-metrics-daemon-f72jl" (UID: "d27497f4-b29e-46c2-8f63-664fbe7e8dd1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:13.722966 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:13.722723 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:13.722966 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:13.722913 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret podName:f947add9-2413-4c26-860a-a5431e7cd2b9 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:29.722898511 +0000 UTC m=+34.271568655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret") pod "global-pull-secret-syncer-9hfm2" (UID: "f947add9-2413-4c26-860a-a5431e7cd2b9") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:13.823410 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:13.823374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwm6\" (UniqueName: \"kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6\") pod \"network-check-target-7lnww\" (UID: \"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe\") " pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:13.823580 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:13.823536 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:16:13.823580 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:13.823556 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:16:13.823580 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:13.823565 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vcwm6 for pod openshift-network-diagnostics/network-check-target-7lnww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:13.823694 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:13.823617 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6 podName:64d8b12c-0fc1-4b11-9a11-8155f18c8dbe nodeName:}" failed. No retries permitted until 2026-10-31 19:16:29.823600349 +0000 UTC m=+34.372270501 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vcwm6" (UniqueName: "kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6") pod "network-check-target-7lnww" (UID: "64d8b12c-0fc1-4b11-9a11-8155f18c8dbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:14.086963 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:14.086891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:14.086963 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:14.086909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:14.086963 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:14.086913 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:14.087185 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:14.086986 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:14.087185 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:14.087081 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:14.087185 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:14.087158 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:15.233377 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:15.233165 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" event={"ID":"f6c9d77e-0f26-4a74-a08b-a2aff00389d2","Type":"ContainerStarted","Data":"8bfe8b200a50dc1b3cc823236a1f7b1b64b7359a209c0be4dfd8ba9445b33551"} Oct 31 19:16:15.233915 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:15.233450 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" event={"ID":"f6c9d77e-0f26-4a74-a08b-a2aff00389d2","Type":"ContainerStarted","Data":"e916020be93088b0f086727601f362b36917aace1bb7a126a2aa81c85ea574d4"} Oct 31 19:16:15.233915 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:15.233525 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" event={"ID":"f6c9d77e-0f26-4a74-a08b-a2aff00389d2","Type":"ContainerStarted","Data":"dbd3f576e7f2a0f2e4130352644dc93521bb94f747376bba385aa18f9a49b60a"} Oct 31 19:16:15.233915 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:15.233556 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" event={"ID":"f6c9d77e-0f26-4a74-a08b-a2aff00389d2","Type":"ContainerStarted","Data":"d219bd56962c9f8e2dbc00aef42e2edc08fad32c196faaa77e2d853994e4999d"} Oct 31 19:16:15.233915 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:15.233585 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" event={"ID":"f6c9d77e-0f26-4a74-a08b-a2aff00389d2","Type":"ContainerStarted","Data":"7c05b3de1240f19465bb4bcb1d42a1336d5ae0c705801d536306862b13f66d08"} Oct 31 19:16:15.233915 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:15.233613 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" event={"ID":"f6c9d77e-0f26-4a74-a08b-a2aff00389d2","Type":"ContainerStarted","Data":"cc5f4f17c0081863724f9e9a7a5d4bcc48e2fd28e93760eb85d5459389f371cf"} Oct 31 19:16:16.088161 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:16.088123 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:16.088350 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:16.088259 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:16.088350 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:16.088307 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:16.088459 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:16.088404 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:16.088459 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:16.088434 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:16.088553 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:16.088493 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:16.237807 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:16.237768 2578 generic.go:356] "Generic (PLEG): container finished" podID="890a45ca-fe6f-4c91-9fda-034924575914" containerID="928096bdea19e27a21c4aaf240f0880c6dc96ee369f63768dd56ddd82afbe424" exitCode=0 Oct 31 19:16:16.237807 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:16.237808 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxslf" event={"ID":"890a45ca-fe6f-4c91-9fda-034924575914","Type":"ContainerDied","Data":"928096bdea19e27a21c4aaf240f0880c6dc96ee369f63768dd56ddd82afbe424"} Oct 31 19:16:17.243441 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:17.243409 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" event={"ID":"f6c9d77e-0f26-4a74-a08b-a2aff00389d2","Type":"ContainerStarted","Data":"87a1e15f35b15be20fe4aa250007f971efcc64384d67941dd8d2843de5d32afc"} Oct 31 19:16:17.246247 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:17.246221 2578 generic.go:356] "Generic (PLEG): container finished" podID="890a45ca-fe6f-4c91-9fda-034924575914" containerID="db55014858bfc8cb3ca1a53377caa819477e60af9d8f8026e75be22f84612329" exitCode=0 Oct 31 19:16:17.246356 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:17.246258 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxslf" event={"ID":"890a45ca-fe6f-4c91-9fda-034924575914","Type":"ContainerDied","Data":"db55014858bfc8cb3ca1a53377caa819477e60af9d8f8026e75be22f84612329"} Oct 31 19:16:18.086779 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:18.086737 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:18.086779 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:18.086766 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:18.086991 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:18.086897 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:18.086991 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:18.086933 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:18.087108 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:18.087081 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:18.087183 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:18.087167 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:18.251458 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:18.251427 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xxslf" event={"ID":"890a45ca-fe6f-4c91-9fda-034924575914","Type":"ContainerStarted","Data":"e8a390ded2c1db7dcf0f303fbd27b2f1d35576169dccf8fc01130f42d98bbfd7"} Oct 31 19:16:18.410855 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:18.410760 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xxslf" podStartSLOduration=8.492788801 podStartE2EDuration="22.410747027s" totalImagesPullingTime="13.917958226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:18.410205614 +0000 UTC m=+22.958875763" watchObservedRunningTime="2026-10-31 19:16:18.410747027 +0000 UTC m=+22.959417174" Oct 31 19:16:19.256644 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:19.256561 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" event={"ID":"f6c9d77e-0f26-4a74-a08b-a2aff00389d2","Type":"ContainerStarted","Data":"42904c8098f03bbbdda9eef57c4a7cfe4e012d0aeda0ea45873690ddf9b959ef"} Oct 31 19:16:19.537499 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:19.537382 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" podStartSLOduration=7.512846161 podStartE2EDuration="23.537370413s" totalImagesPullingTime="16.024524252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:19.489794632 +0000 UTC m=+24.038464781" watchObservedRunningTime="2026-10-31 19:16:19.537370413 +0000 UTC m=+24.086040561" Oct 31 19:16:20.087324 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:20.087290 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:20.087503 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:20.087290 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:20.087503 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:20.087427 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:20.087503 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:20.087290 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:20.087503 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:20.087474 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:20.087718 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:20.087564 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:20.258984 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:20.258943 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:16:20.259406 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:20.259026 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:16:20.259406 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:20.259072 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:16:20.274081 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:20.274054 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:16:20.274227 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:20.274166 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:16:21.438055 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:21.437950 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7lnww"] Oct 31 19:16:21.438426 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:21.438067 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:21.438426 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:21.438162 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:21.441495 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:21.441465 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9hfm2"] Oct 31 19:16:21.441643 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:21.441575 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:21.441687 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:21.441649 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:21.442106 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:21.442088 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f72jl"] Oct 31 19:16:21.442170 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:21.442160 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:21.442240 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:21.442227 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:23.087130 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:23.087098 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:23.087559 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:23.087181 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:23.087559 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:23.087202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:23.087559 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:23.087316 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:23.087559 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:23.087391 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:23.087559 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:23.087446 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:25.087391 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:25.087308 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:25.087866 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:25.087420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:25.087866 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:25.087433 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:25.087866 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:25.087445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:25.087866 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:25.087504 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:25.087866 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:25.087584 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:25.783391 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:25.783364 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h89v4_0875c4d7-7bfd-45ab-ae34-75e20e327605/dns-node-resolver/0.log" Oct 31 19:16:26.619568 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:26.619541 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dzg4p_9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f/node-ca/0.log" Oct 31 19:16:27.087303 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:27.087273 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:27.087303 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:27.087303 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:27.087303 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:27.087303 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:27.087590 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:27.087397 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:27.087590 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:27.087513 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:27.087590 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:27.087576 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:29.086696 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:29.086658 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:29.087139 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:29.086754 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:29.087139 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:29.086772 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:29.087139 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:29.086773 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:29.087139 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:29.086843 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:29.087139 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:29.086926 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:29.733902 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:29.733854 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:29.733902 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:29.733902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:29.734149 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:29.733994 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:29.734149 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:29.734052 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs podName:d27497f4-b29e-46c2-8f63-664fbe7e8dd1 nodeName:}" failed. No retries permitted until 2026-10-31 19:17:01.734034907 +0000 UTC m=+66.282705038 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs") pod "network-metrics-daemon-f72jl" (UID: "d27497f4-b29e-46c2-8f63-664fbe7e8dd1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:29.734149 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:29.733992 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:29.734149 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:29.734131 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret podName:f947add9-2413-4c26-860a-a5431e7cd2b9 nodeName:}" failed. No retries permitted until 2026-10-31 19:17:01.734116188 +0000 UTC m=+66.282786332 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret") pod "global-pull-secret-syncer-9hfm2" (UID: "f947add9-2413-4c26-860a-a5431e7cd2b9") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:29.835199 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:29.835165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwm6\" (UniqueName: \"kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6\") pod \"network-check-target-7lnww\" (UID: \"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe\") " pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:29.835562 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:29.835543 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:16:29.835680 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:29.835664 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:16:29.835755 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:29.835747 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vcwm6 for pod openshift-network-diagnostics/network-check-target-7lnww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:29.835876 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:29.835865 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6 podName:64d8b12c-0fc1-4b11-9a11-8155f18c8dbe nodeName:}" failed. No retries permitted until 2026-10-31 19:17:01.835839509 +0000 UTC m=+66.384509644 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vcwm6" (UniqueName: "kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6") pod "network-check-target-7lnww" (UID: "64d8b12c-0fc1-4b11-9a11-8155f18c8dbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:31.087335 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:31.087309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:31.087670 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:31.087373 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:31.087670 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:31.087401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:31.087670 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:31.087473 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:31.087670 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:31.087562 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:31.087670 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:31.087648 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:33.087116 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:33.087088 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:33.087116 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:33.087113 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:33.087603 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:33.087204 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:33.087603 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:33.087242 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:33.087603 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:33.087327 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:33.087603 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:33.087408 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:35.087062 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:35.087025 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:35.087062 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:35.087050 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:35.087504 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:35.087135 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:35.087504 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:35.087144 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:35.087504 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:35.087258 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:35.087504 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:35.087358 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:37.087331 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:37.087307 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:37.087706 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:37.087408 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:37.087706 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:37.087422 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:37.087706 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:37.087480 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:37.087706 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:37.087499 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:37.087706 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:37.087594 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:39.087491 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.087398 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:39.087491 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.087467 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:39.087930 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.087493 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:39.087930 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:39.087576 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lnww" podUID="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" Oct 31 19:16:39.087930 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:39.087647 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hfm2" podUID="f947add9-2413-4c26-860a-a5431e7cd2b9" Oct 31 19:16:39.087930 ip-10-0-128-164 kubenswrapper[2578]: E1031 19:16:39.087726 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72jl" podUID="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" Oct 31 19:16:39.208858 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.208833 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeReady" Oct 31 19:16:39.209020 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.208961 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 19:16:39.356165 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.356088 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4kmqj"] Oct 31 19:16:39.388616 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.388591 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zxkn8"] Oct 31 19:16:39.388794 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.388700 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4kmqj" Oct 31 19:16:39.395166 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.395139 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z4stw"] Oct 31 19:16:39.395300 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.395266 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.417407 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.417379 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4kmqj"] Oct 31 19:16:39.417521 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.417416 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z4stw"] Oct 31 19:16:39.417521 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.417437 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zxkn8"] Oct 31 19:16:39.417521 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.417502 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.445762 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.445737 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 19:16:39.445762 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.445751 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 19:16:39.496724 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.496700 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgzhm\"" Oct 31 19:16:39.497069 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.497044 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 19:16:39.497069 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.497044 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mcs56\"" Oct 31 19:16:39.497205 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.497089 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 19:16:39.497205 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.497153 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4b89l\"" Oct 31 19:16:39.497205 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.497203 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 19:16:39.506317 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.506296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z96kc\" (UniqueName: \"kubernetes.io/projected/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-kube-api-access-z96kc\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.506366 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.506328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-metrics-tls\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.506396 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.506362 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d851015-b321-4b76-b09f-45c307ecfb37-crio-socket\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.506465 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.506452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d851015-b321-4b76-b09f-45c307ecfb37-data-volume\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.506503 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.506477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d851015-b321-4b76-b09f-45c307ecfb37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.506503 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.506494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-tmp-dir\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.506558 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.506519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwl2\" (UniqueName: \"kubernetes.io/projected/a302be0c-ed10-4c2d-8673-debd526d4994-kube-api-access-gtwl2\") pod \"ingress-canary-4kmqj\" (UID: \"a302be0c-ed10-4c2d-8673-debd526d4994\") " pod="openshift-ingress-canary/ingress-canary-4kmqj" Oct 31 19:16:39.506609 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.506597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d851015-b321-4b76-b09f-45c307ecfb37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.506648 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.506637 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-config-volume\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.506706 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.506690 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r6m7\" (UniqueName: \"kubernetes.io/projected/2d851015-b321-4b76-b09f-45c307ecfb37-kube-api-access-7r6m7\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.506749 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.506737 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a302be0c-ed10-4c2d-8673-debd526d4994-cert\") pod \"ingress-canary-4kmqj\" (UID: \"a302be0c-ed10-4c2d-8673-debd526d4994\") " pod="openshift-ingress-canary/ingress-canary-4kmqj" Oct 31 19:16:39.546576 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.546555 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 19:16:39.546675 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.546598 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 19:16:39.546675 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.546602 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 19:16:39.546873 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.546856 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 19:16:39.607154 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r6m7\" (UniqueName: \"kubernetes.io/projected/2d851015-b321-4b76-b09f-45c307ecfb37-kube-api-access-7r6m7\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.607154 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a302be0c-ed10-4c2d-8673-debd526d4994-cert\") pod \"ingress-canary-4kmqj\" (UID: \"a302be0c-ed10-4c2d-8673-debd526d4994\") " pod="openshift-ingress-canary/ingress-canary-4kmqj" Oct 31 19:16:39.607154 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z96kc\" (UniqueName: \"kubernetes.io/projected/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-kube-api-access-z96kc\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.607306 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-metrics-tls\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.607306 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d851015-b321-4b76-b09f-45c307ecfb37-crio-socket\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.607306 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d851015-b321-4b76-b09f-45c307ecfb37-data-volume\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.607306 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d851015-b321-4b76-b09f-45c307ecfb37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.607306 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-tmp-dir\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.607306 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwl2\" (UniqueName: \"kubernetes.io/projected/a302be0c-ed10-4c2d-8673-debd526d4994-kube-api-access-gtwl2\") pod \"ingress-canary-4kmqj\" (UID: \"a302be0c-ed10-4c2d-8673-debd526d4994\") " pod="openshift-ingress-canary/ingress-canary-4kmqj" Oct 31 19:16:39.607600 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607410 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d851015-b321-4b76-b09f-45c307ecfb37-crio-socket\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.607659 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d851015-b321-4b76-b09f-45c307ecfb37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.607659 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-config-volume\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.607659 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607645 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-tmp-dir\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.607818 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.607797 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d851015-b321-4b76-b09f-45c307ecfb37-data-volume\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.608118 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.608101 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-config-volume\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.608182 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.608115 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d851015-b321-4b76-b09f-45c307ecfb37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.610732 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.610706 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-metrics-tls\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.610852 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.610758 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d851015-b321-4b76-b09f-45c307ecfb37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.610962 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.610944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a302be0c-ed10-4c2d-8673-debd526d4994-cert\") pod \"ingress-canary-4kmqj\" (UID: \"a302be0c-ed10-4c2d-8673-debd526d4994\") " pod="openshift-ingress-canary/ingress-canary-4kmqj" Oct 31 19:16:39.661034 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.661010 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r6m7\" (UniqueName: \"kubernetes.io/projected/2d851015-b321-4b76-b09f-45c307ecfb37-kube-api-access-7r6m7\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.661137 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.661040 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z96kc\" (UniqueName: \"kubernetes.io/projected/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-kube-api-access-z96kc\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:39.661808 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.661792 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwl2\" (UniqueName: \"kubernetes.io/projected/a302be0c-ed10-4c2d-8673-debd526d4994-kube-api-access-gtwl2\") pod \"ingress-canary-4kmqj\" (UID: \"a302be0c-ed10-4c2d-8673-debd526d4994\") " pod="openshift-ingress-canary/ingress-canary-4kmqj" Oct 31 19:16:39.718296 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.718269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4kmqj" Oct 31 19:16:39.765738 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.765712 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:16:39.791807 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:39.791785 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:40.199428 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:40.199401 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4kmqj"] Oct 31 19:16:40.220545 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:40.220526 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z4stw"] Oct 31 19:16:40.221562 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:40.221528 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zxkn8"] Oct 31 19:16:40.222672 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:16:40.222647 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c680bdb_3c36_4584_b1ec_78ab49d07aa9.slice/crio-3647333a5633307897889f185158cebaf5d49c9c638f0b932a8cac97cf23439f WatchSource:0}: Error finding container 3647333a5633307897889f185158cebaf5d49c9c638f0b932a8cac97cf23439f: Status 404 returned error can't find the container with id 3647333a5633307897889f185158cebaf5d49c9c638f0b932a8cac97cf23439f Oct 31 19:16:40.233704 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:16:40.233684 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d851015_b321_4b76_b09f_45c307ecfb37.slice/crio-fc92348b50a855925ee720198d8ab164f99f78c91b179b19eae08713e3876371 WatchSource:0}: Error finding container fc92348b50a855925ee720198d8ab164f99f78c91b179b19eae08713e3876371: Status 404 returned error can't find the container with id fc92348b50a855925ee720198d8ab164f99f78c91b179b19eae08713e3876371 Oct 31 19:16:40.301221 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:40.301191 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zxkn8" event={"ID":"2d851015-b321-4b76-b09f-45c307ecfb37","Type":"ContainerStarted","Data":"fc92348b50a855925ee720198d8ab164f99f78c91b179b19eae08713e3876371"} Oct 31 19:16:40.302412 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:40.302387 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4kmqj" event={"ID":"a302be0c-ed10-4c2d-8673-debd526d4994","Type":"ContainerStarted","Data":"595187ad7830077771218255120ad2563a91877595b00108b32f7f237fa734d2"} Oct 31 19:16:40.303487 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:40.303466 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4stw" event={"ID":"9c680bdb-3c36-4584-b1ec-78ab49d07aa9","Type":"ContainerStarted","Data":"3647333a5633307897889f185158cebaf5d49c9c638f0b932a8cac97cf23439f"} Oct 31 19:16:41.087576 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:41.087549 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:16:41.087576 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:41.087572 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:16:41.087815 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:41.087731 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:16:41.137965 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:41.137945 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 19:16:41.137965 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:41.137943 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 19:16:41.185461 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:41.185428 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qh5p\"" Oct 31 19:16:41.185461 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:41.185460 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tgkwh\"" Oct 31 19:16:41.185654 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:41.185482 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgqq8\"" Oct 31 19:16:41.185654 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:41.185490 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 19:16:41.185654 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:41.185486 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 19:16:41.307285 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:41.307250 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zxkn8" event={"ID":"2d851015-b321-4b76-b09f-45c307ecfb37","Type":"ContainerStarted","Data":"72e29ef52f6d0aa0d850a76c5b619ba195aa02af9faffbf8ae17929d2d0e5760"} Oct 31 19:16:41.833490 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:41.833463 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h89v4_0875c4d7-7bfd-45ab-ae34-75e20e327605/dns-node-resolver/0.log" Oct 31 19:16:42.311116 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:42.311079 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zxkn8" event={"ID":"2d851015-b321-4b76-b09f-45c307ecfb37","Type":"ContainerStarted","Data":"0e88cbcc40cada77955c68e1e3f909d41101bfb4c1892cb7d32b30eeb2cf12cb"} Oct 31 19:16:43.034099 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:43.034072 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dzg4p_9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f/node-ca/0.log" Oct 31 19:16:44.318026 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:44.317987 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4stw" event={"ID":"9c680bdb-3c36-4584-b1ec-78ab49d07aa9","Type":"ContainerStarted","Data":"5b751cf756b4acdbf17087e7f6efb8be5625321c3a8b30a9bcef9d25501570f8"} Oct 31 19:16:44.318387 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:44.318052 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4stw" event={"ID":"9c680bdb-3c36-4584-b1ec-78ab49d07aa9","Type":"ContainerStarted","Data":"8f75b95e10ce549983c46f2bd749594c88a5ee17f71aa925cefa5e0333ca5b21"} Oct 31 19:16:44.318387 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:44.318083 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:44.473362 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:44.473306 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-z4stw" podStartSLOduration=2.50047644 podStartE2EDuration="5.473287629s" totalImagesPullingTime="2.972811189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:44.472990103 +0000 UTC m=+49.021660250" watchObservedRunningTime="2026-10-31 19:16:44.473287629 +0000 UTC m=+49.021957780" Oct 31 19:16:44.584484 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:44.584385 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zxkn8" podStartSLOduration=1.754950348 podStartE2EDuration="5.584365497s" totalImagesPullingTime="3.829415149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:16:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:44.583241103 +0000 UTC m=+49.131911251" watchObservedRunningTime="2026-10-31 19:16:44.584365497 +0000 UTC m=+49.133035645" Oct 31 19:16:45.324322 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:45.324287 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zxkn8" event={"ID":"2d851015-b321-4b76-b09f-45c307ecfb37","Type":"ContainerStarted","Data":"e43123acec66a3733b5809287f96c23e1be93e1200833b03a4aa1e57c931e1c1"} Oct 31 19:16:45.325749 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:45.325722 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4kmqj" event={"ID":"a302be0c-ed10-4c2d-8673-debd526d4994","Type":"ContainerStarted","Data":"374a370afb6c12b96aa5425df0554d8a6826f1a6619dcb6ae682cd76718302e1"} Oct 31 19:16:45.437335 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:45.437292 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4kmqj" podStartSLOduration=1.851148299 podStartE2EDuration="6.437279304s" totalImagesPullingTime="4.586131005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:45.436492608 +0000 UTC m=+49.985162756" watchObservedRunningTime="2026-10-31 19:16:45.437279304 +0000 UTC m=+49.985949487" Oct 31 19:16:48.913627 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:48.913590 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s"] Oct 31 19:16:48.915823 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:48.915808 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:48.960890 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:48.960863 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f"] Oct 31 19:16:48.962836 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:48.962821 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:48.966437 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:48.966420 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 19:16:49.007517 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.007492 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 19:16:49.007517 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.007510 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-m972p\"" Oct 31 19:16:49.007667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.007522 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 19:16:49.007667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.007515 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 19:16:49.012591 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.012569 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 19:16:49.013728 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.013711 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s"] Oct 31 19:16:49.014335 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.014319 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f"] Oct 31 19:16:49.043701 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.043681 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2hfw5"] Oct 31 19:16:49.045756 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.045742 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.066756 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.066730 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 19:16:49.067136 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.067118 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rqvd7\"" Oct 31 19:16:49.067213 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.067120 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 19:16:49.067259 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.067207 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 19:16:49.085642 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.085618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6310b145-4c78-4f58-91d0-be7558e3b1cc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.085723 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.085672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rps8q\" (UniqueName: \"kubernetes.io/projected/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-api-access-rps8q\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.085761 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.085737 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.085818 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.085799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6310b145-4c78-4f58-91d0-be7558e3b1cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.085853 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.085831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6310b145-4c78-4f58-91d0-be7558e3b1cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.085908 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.085874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.085944 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.085909 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.085944 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.085925 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr4k7\" (UniqueName: \"kubernetes.io/projected/6310b145-4c78-4f58-91d0-be7558e3b1cc-kube-api-access-xr4k7\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.086002 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.085987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.086031 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.086007 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.166081 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.166004 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mvdsq\"" Oct 31 19:16:49.166220 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.166195 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 19:16:49.166275 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.166236 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 19:16:49.166356 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.166339 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 19:16:49.186976 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.186953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-sys\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.187095 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.186983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.187095 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.187195 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.187195 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187124 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-textfile\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.187195 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqsxp\" (UniqueName: \"kubernetes.io/projected/6d131de0-3a07-4d70-99e7-7aedb4648f31-kube-api-access-pqsxp\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.187195 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d131de0-3a07-4d70-99e7-7aedb4648f31-metrics-client-ca\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.187195 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6310b145-4c78-4f58-91d0-be7558e3b1cc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.187195 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-tls\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.187466 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-wtmp\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.187466 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.187466 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187446 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.187466 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rps8q\" (UniqueName: \"kubernetes.io/projected/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-api-access-rps8q\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.187644 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.187644 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187532 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-root\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.187644 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr4k7\" (UniqueName: \"kubernetes.io/projected/6310b145-4c78-4f58-91d0-be7558e3b1cc-kube-api-access-xr4k7\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.187644 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-accelerators-collector-config\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.187644 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6310b145-4c78-4f58-91d0-be7558e3b1cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.187874 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.187874 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6310b145-4c78-4f58-91d0-be7558e3b1cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.188002 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.187939 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6310b145-4c78-4f58-91d0-be7558e3b1cc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.188050 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.188023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.188296 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.188274 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.189984 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.189962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6310b145-4c78-4f58-91d0-be7558e3b1cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.190094 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.190058 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.190169 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.190146 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.190310 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.190295 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6310b145-4c78-4f58-91d0-be7558e3b1cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.246930 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.246902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr4k7\" (UniqueName: \"kubernetes.io/projected/6310b145-4c78-4f58-91d0-be7558e3b1cc-kube-api-access-xr4k7\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.247041 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.246932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rps8q\" (UniqueName: \"kubernetes.io/projected/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-api-access-rps8q\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.274384 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.274349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:16:49.288959 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.288932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-accelerators-collector-config\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289072 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.288969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-sys\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289072 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-sys\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289072 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-textfile\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289226 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqsxp\" (UniqueName: \"kubernetes.io/projected/6d131de0-3a07-4d70-99e7-7aedb4648f31-kube-api-access-pqsxp\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289226 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d131de0-3a07-4d70-99e7-7aedb4648f31-metrics-client-ca\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289226 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-tls\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289226 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-wtmp\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289226 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289226 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-root\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289488 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289322 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-root\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289488 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289341 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-textfile\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289488 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289461 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-wtmp\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289658 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289544 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-accelerators-collector-config\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.289658 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.289650 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d131de0-3a07-4d70-99e7-7aedb4648f31-metrics-client-ca\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.291455 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.291430 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-tls\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.291559 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.291531 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.342205 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.342177 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqsxp\" (UniqueName: \"kubernetes.io/projected/6d131de0-3a07-4d70-99e7-7aedb4648f31-kube-api-access-pqsxp\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.358198 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.358177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:16:49.368984 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:16:49.368960 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d131de0_3a07_4d70_99e7_7aedb4648f31.slice/crio-76d1cc3979571efddf33afe750f327b28d66c780f5e27b637f0873adfc6d2c6f WatchSource:0}: Error finding container 76d1cc3979571efddf33afe750f327b28d66c780f5e27b637f0873adfc6d2c6f: Status 404 returned error can't find the container with id 76d1cc3979571efddf33afe750f327b28d66c780f5e27b637f0873adfc6d2c6f Oct 31 19:16:49.527608 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.527579 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:16:49.677532 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.677505 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f"] Oct 31 19:16:49.679782 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:16:49.679752 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1441c08_c76a_4954_80c9_8c1a0a04ce9f.slice/crio-5b92c0b3aba96f21982c7ad2c7aaf06d4f29788cc48c9f073e004531f51e3b04 WatchSource:0}: Error finding container 5b92c0b3aba96f21982c7ad2c7aaf06d4f29788cc48c9f073e004531f51e3b04: Status 404 returned error can't find the container with id 5b92c0b3aba96f21982c7ad2c7aaf06d4f29788cc48c9f073e004531f51e3b04 Oct 31 19:16:49.887560 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:49.887482 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s"] Oct 31 19:16:49.889701 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:16:49.889679 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6310b145_4c78_4f58_91d0_be7558e3b1cc.slice/crio-1b83d6279ed781dee8360c7d1fb625aa60d82cab0349f98742c8d053afcb889c WatchSource:0}: Error finding container 1b83d6279ed781dee8360c7d1fb625aa60d82cab0349f98742c8d053afcb889c: Status 404 returned error can't find the container with id 1b83d6279ed781dee8360c7d1fb625aa60d82cab0349f98742c8d053afcb889c Oct 31 19:16:50.340027 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:50.339992 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2hfw5" event={"ID":"6d131de0-3a07-4d70-99e7-7aedb4648f31","Type":"ContainerStarted","Data":"76d1cc3979571efddf33afe750f327b28d66c780f5e27b637f0873adfc6d2c6f"} Oct 31 19:16:50.341822 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:50.341797 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" event={"ID":"6310b145-4c78-4f58-91d0-be7558e3b1cc","Type":"ContainerStarted","Data":"6f3aeb516c67716502ac39ffdfbddec037301c3354b698dc949cff2ea38b8e38"} Oct 31 19:16:50.341972 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:50.341847 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" event={"ID":"6310b145-4c78-4f58-91d0-be7558e3b1cc","Type":"ContainerStarted","Data":"c5cdf282dafc2546ff11b71e4149ff94f6399a249ac1bec9418b5b397ffd472f"} Oct 31 19:16:50.341972 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:50.341902 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" event={"ID":"6310b145-4c78-4f58-91d0-be7558e3b1cc","Type":"ContainerStarted","Data":"1b83d6279ed781dee8360c7d1fb625aa60d82cab0349f98742c8d053afcb889c"} Oct 31 19:16:50.343003 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:50.342982 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" event={"ID":"e1441c08-c76a-4954-80c9-8c1a0a04ce9f","Type":"ContainerStarted","Data":"5b92c0b3aba96f21982c7ad2c7aaf06d4f29788cc48c9f073e004531f51e3b04"} Oct 31 19:16:51.290784 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.289142 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 19:16:51.292133 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.292107 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtq5n\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-kube-api-access-mtq5n\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-config-volume\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-config-out\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308608 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-web-config\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.308943 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.308718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.339742 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.339712 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 19:16:51.349319 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.349285 2578 generic.go:356] "Generic (PLEG): container finished" podID="6d131de0-3a07-4d70-99e7-7aedb4648f31" containerID="d2acaf63d0793f9762c91720ff2afe25d2344656a98637fb3a366bfe3756c3fa" exitCode=0 Oct 31 19:16:51.349712 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.349358 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2hfw5" event={"ID":"6d131de0-3a07-4d70-99e7-7aedb4648f31","Type":"ContainerDied","Data":"d2acaf63d0793f9762c91720ff2afe25d2344656a98637fb3a366bfe3756c3fa"} Oct 31 19:16:51.385431 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.385392 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mhxbx\"" Oct 31 19:16:51.385612 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.385473 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 19:16:51.385612 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.385558 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 19:16:51.385717 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.385643 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 19:16:51.385717 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.385409 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 19:16:51.385822 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.385803 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 19:16:51.385822 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.385814 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 19:16:51.385923 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.385873 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 19:16:51.410015 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.409984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.410100 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.410028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq5n\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-kube-api-access-mtq5n\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.410100 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.410059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.410100 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.410084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-config-volume\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.410222 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.410108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-config-out\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.410222 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.410130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.410222 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.410154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.410357 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.410224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.410357 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.410260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.410357 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.410288 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-web-config\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.410504 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.410362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.410504 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.410389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.410504 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.410471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.411969 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.411826 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.412584 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.412305 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.413225 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.413200 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.414670 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.414638 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.415846 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.415806 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-config-out\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.415961 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.415940 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.416701 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.416660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-web-config\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.419122 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.418571 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.419122 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.419087 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.419245 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.419094 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-config-volume\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.437498 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.437469 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 19:16:51.443164 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.443108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.443373 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.443355 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 19:16:51.455237 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.455211 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.483869 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.483846 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtq5n\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-kube-api-access-mtq5n\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:51.616253 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:51.615983 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:16:52.037427 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:52.037383 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 19:16:52.039383 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:16:52.039355 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25720c7c_d528_4902_a94a_f437589ae9c9.slice/crio-40f3088e96e3723af2d48668d39a9bc14af3f66038c105289ef1a00eaf7d60b8 WatchSource:0}: Error finding container 40f3088e96e3723af2d48668d39a9bc14af3f66038c105289ef1a00eaf7d60b8: Status 404 returned error can't find the container with id 40f3088e96e3723af2d48668d39a9bc14af3f66038c105289ef1a00eaf7d60b8 Oct 31 19:16:52.274293 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:52.274267 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:16:52.354844 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:52.354803 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2hfw5" event={"ID":"6d131de0-3a07-4d70-99e7-7aedb4648f31","Type":"ContainerStarted","Data":"a824ec4f6a8d672e030cfb8036add31d21fc6b644650fb585f5c601fb9b95c66"} Oct 31 19:16:52.355290 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:52.354916 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2hfw5" event={"ID":"6d131de0-3a07-4d70-99e7-7aedb4648f31","Type":"ContainerStarted","Data":"5ed986559b3f28442185d71f066f040b6e303363efcc91e5ae134fcc9774a742"} Oct 31 19:16:52.356778 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:52.356756 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" event={"ID":"6310b145-4c78-4f58-91d0-be7558e3b1cc","Type":"ContainerStarted","Data":"0be4a567499d4320fe474640ae1490bf0c7258e3bdf82e5d9e436d2b088bda5b"} Oct 31 19:16:52.358828 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:52.358809 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" event={"ID":"e1441c08-c76a-4954-80c9-8c1a0a04ce9f","Type":"ContainerStarted","Data":"94d02c53a08992845fefc94560f1d05c36d2fac6c944d9f6ba31753b557b1f6b"} Oct 31 19:16:52.358906 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:52.358854 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" event={"ID":"e1441c08-c76a-4954-80c9-8c1a0a04ce9f","Type":"ContainerStarted","Data":"05386f176d2f067372a59d518a4b434f82697c0831a700bbddee6b0fd52241c3"} Oct 31 19:16:52.358906 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:52.358876 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" event={"ID":"e1441c08-c76a-4954-80c9-8c1a0a04ce9f","Type":"ContainerStarted","Data":"b90de1a6318d6c46955c6ccc8ad14dc5d7fe2b2a43ba1a02bf9b447f2794ccb9"} Oct 31 19:16:52.359900 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:52.359866 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerStarted","Data":"40f3088e96e3723af2d48668d39a9bc14af3f66038c105289ef1a00eaf7d60b8"} Oct 31 19:16:52.532213 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:52.532094 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2hfw5" podStartSLOduration=2.949164445 podStartE2EDuration="4.532080437s" totalImagesPullingTime="1.517150982s" totalInitContainerRuntime="65.76501ms" isStatefulPod=true podCreationTimestamp="2026-10-31 19:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:52.531512435 +0000 UTC m=+57.080182583" watchObservedRunningTime="2026-10-31 19:16:52.532080437 +0000 UTC m=+57.080750581" Oct 31 19:16:52.643455 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:52.643402 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" podStartSLOduration=2.915328936 podStartE2EDuration="4.643388182s" totalImagesPullingTime="1.728059246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:52.641627189 +0000 UTC m=+57.190297340" watchObservedRunningTime="2026-10-31 19:16:52.643388182 +0000 UTC m=+57.192058348" Oct 31 19:16:52.754327 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:52.754275 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" podStartSLOduration=2.608955946 podStartE2EDuration="4.754260272s" totalImagesPullingTime="2.145304326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:52.753333151 +0000 UTC m=+57.302003298" watchObservedRunningTime="2026-10-31 19:16:52.754260272 +0000 UTC m=+57.302930421" Oct 31 19:16:54.328536 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:54.328506 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z4stw" Oct 31 19:16:54.368389 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:54.368362 2578 generic.go:356] "Generic (PLEG): container finished" podID="25720c7c-d528-4902-a94a-f437589ae9c9" containerID="143614d224bcde8e6b9dd3dcecb9d5a2987446d1c10395305c426a181efc5f25" exitCode=0 Oct 31 19:16:54.368559 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:54.368415 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerDied","Data":"143614d224bcde8e6b9dd3dcecb9d5a2987446d1c10395305c426a181efc5f25"} Oct 31 19:16:57.380845 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:57.380809 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerStarted","Data":"6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85"} Oct 31 19:16:57.381274 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:57.380921 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerStarted","Data":"02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72"} Oct 31 19:16:57.381274 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:57.380948 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerStarted","Data":"4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5"} Oct 31 19:16:57.381274 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:57.380971 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerStarted","Data":"28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1"} Oct 31 19:16:57.381274 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:57.380993 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerStarted","Data":"782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8"} Oct 31 19:16:57.972206 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:57.972176 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 19:16:57.975046 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:57.975026 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.025520 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.025476 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 19:16:58.025687 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.025539 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 19:16:58.025687 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.025553 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 19:16:58.025772 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.025751 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 19:16:58.025831 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.025785 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 19:16:58.072814 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.072779 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 19:16:58.072814 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.072789 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 19:16:58.073052 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.072820 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 19:16:58.073052 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.072824 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 19:16:58.073139 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073056 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c47hb7itrpuc4\"" Oct 31 19:16:58.073139 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073066 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 19:16:58.073139 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073117 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 19:16:58.073327 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073140 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qq6mx\"" Oct 31 19:16:58.073445 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073427 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 19:16:58.073581 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.073658 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.073716 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.073716 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj4gh\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-kube-api-access-vj4gh\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.073812 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.073812 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.073942 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073820 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.073942 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.073942 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.073942 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073918 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-web-config\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.074124 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.074124 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073973 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.074124 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.073997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.074124 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.074022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.074124 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.074055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-config\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.074124 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.074087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.074399 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.074144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.074399 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.074170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.074399 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.074194 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.074399 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.074228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-config-out\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.120988 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.120960 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 19:16:58.120988 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.120962 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 19:16:58.122485 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.122463 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 19:16:58.168475 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.168447 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 19:16:58.175607 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.175583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.175739 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.175619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-web-config\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.175739 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.175651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.175868 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.175771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.175868 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.175817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.175868 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.175856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176038 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.175905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-config\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176038 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.175935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176038 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.175983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176038 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176038 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176272 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-config-out\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176272 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176272 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176272 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176272 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4gh\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-kube-api-access-vj4gh\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176546 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176546 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176546 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176546 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176730 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176615 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176730 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176643 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176730 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176681 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.176730 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.176710 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.177177 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.177156 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.178039 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.178012 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.182316 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.181359 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-config\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.182316 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.181458 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.182316 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.181490 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.182316 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.181621 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.182316 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.181791 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.182316 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.181874 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-config-out\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.182316 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.181907 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.182316 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.182071 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-web-config\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.182316 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.182222 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.182316 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.182277 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.183496 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.183455 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.183672 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.183653 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.184049 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.184011 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.232390 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.232326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4gh\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-kube-api-access-vj4gh\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.293298 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.293267 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:16:58.387224 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.387191 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerStarted","Data":"11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c"} Oct 31 19:16:58.507138 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.507044 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.23713077 podStartE2EDuration="7.507031405s" totalImagesPullingTime="5.257499357s" totalInitContainerRuntime="12.401278ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:58.505795007 +0000 UTC m=+63.054465156" watchObservedRunningTime="2026-10-31 19:16:58.507031405 +0000 UTC m=+63.055701592" Oct 31 19:16:58.704246 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:58.704210 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 19:16:58.707054 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:16:58.707025 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c88061_9a64_4653_96d8_e40f15c7f60c.slice/crio-8fc52594fbc580ce4ed493bdf7ec0ab5245602e6c62f8c43cfad99ebd61a70b3 WatchSource:0}: Error finding container 8fc52594fbc580ce4ed493bdf7ec0ab5245602e6c62f8c43cfad99ebd61a70b3: Status 404 returned error can't find the container with id 8fc52594fbc580ce4ed493bdf7ec0ab5245602e6c62f8c43cfad99ebd61a70b3 Oct 31 19:16:59.391667 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:59.391637 2578 generic.go:356] "Generic (PLEG): container finished" podID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerID="12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58" exitCode=0 Oct 31 19:16:59.392158 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:59.391728 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerDied","Data":"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58"} Oct 31 19:16:59.392158 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:16:59.391820 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerStarted","Data":"8fc52594fbc580ce4ed493bdf7ec0ab5245602e6c62f8c43cfad99ebd61a70b3"} Oct 31 19:17:01.819251 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:01.819209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:17:01.819729 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:01.819368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:17:01.868126 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:01.868093 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 19:17:01.868126 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:01.868116 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 19:17:01.872614 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:01.872588 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:17:01.872734 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:01.872660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:17:01.920651 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:01.920619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwm6\" (UniqueName: \"kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6\") pod \"network-check-target-7lnww\" (UID: \"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe\") " pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:17:01.922390 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:01.922359 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qh5p\"" Oct 31 19:17:01.924410 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:01.924393 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:17:01.973426 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:01.973393 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 19:17:02.029875 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:02.029847 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 19:17:02.034557 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:02.034523 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcwm6\" (UniqueName: \"kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6\") pod \"network-check-target-7lnww\" (UID: \"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe\") " pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:17:02.166477 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:02.166394 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tgkwh\"" Oct 31 19:17:02.168490 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:02.168469 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:17:02.182950 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:02.182906 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgqq8\"" Oct 31 19:17:02.184866 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:02.184841 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:17:02.298320 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:02.298294 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f72jl"] Oct 31 19:17:02.301135 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:17:02.301100 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27497f4_b29e_46c2_8f63_664fbe7e8dd1.slice/crio-39876f5809d577b25073f5356d4f81793dcbe817779f869fb428725a1fd6b68e WatchSource:0}: Error finding container 39876f5809d577b25073f5356d4f81793dcbe817779f869fb428725a1fd6b68e: Status 404 returned error can't find the container with id 39876f5809d577b25073f5356d4f81793dcbe817779f869fb428725a1fd6b68e Oct 31 19:17:02.404073 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:02.404032 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f72jl" event={"ID":"d27497f4-b29e-46c2-8f63-664fbe7e8dd1","Type":"ContainerStarted","Data":"39876f5809d577b25073f5356d4f81793dcbe817779f869fb428725a1fd6b68e"} Oct 31 19:17:02.534987 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:02.534952 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7lnww"] Oct 31 19:17:02.538484 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:17:02.538455 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d8b12c_0fc1_4b11_9a11_8155f18c8dbe.slice/crio-ca8eeef73881173f6f6d5579bc69430c6f9eeda3671cde33734be427a5bec078 WatchSource:0}: Error finding container ca8eeef73881173f6f6d5579bc69430c6f9eeda3671cde33734be427a5bec078: Status 404 returned error can't find the container with id ca8eeef73881173f6f6d5579bc69430c6f9eeda3671cde33734be427a5bec078 Oct 31 19:17:02.580627 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:02.580604 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9hfm2"] Oct 31 19:17:02.582453 ip-10-0-128-164 kubenswrapper[2578]: W1031 19:17:02.582425 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf947add9_2413_4c26_860a_a5431e7cd2b9.slice/crio-33d65251ad167927b3f30e95c3c44a82f4e1c248f792d13bcbb6a27ef5e904db WatchSource:0}: Error finding container 33d65251ad167927b3f30e95c3c44a82f4e1c248f792d13bcbb6a27ef5e904db: Status 404 returned error can't find the container with id 33d65251ad167927b3f30e95c3c44a82f4e1c248f792d13bcbb6a27ef5e904db Oct 31 19:17:03.409002 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:03.408924 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9hfm2" event={"ID":"f947add9-2413-4c26-860a-a5431e7cd2b9","Type":"ContainerStarted","Data":"33d65251ad167927b3f30e95c3c44a82f4e1c248f792d13bcbb6a27ef5e904db"} Oct 31 19:17:03.410502 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:03.410476 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7lnww" event={"ID":"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe","Type":"ContainerStarted","Data":"ca8eeef73881173f6f6d5579bc69430c6f9eeda3671cde33734be427a5bec078"} Oct 31 19:17:03.412852 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:03.412828 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerStarted","Data":"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b"} Oct 31 19:17:03.412959 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:03.412899 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerStarted","Data":"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b"} Oct 31 19:17:04.420977 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:04.420933 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f72jl" event={"ID":"d27497f4-b29e-46c2-8f63-664fbe7e8dd1","Type":"ContainerStarted","Data":"965d57aa4ff7ce7ef88a9cbd87915a817a1c613ea82faf015100535b3ad18634"} Oct 31 19:17:04.421426 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:04.421004 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f72jl" event={"ID":"d27497f4-b29e-46c2-8f63-664fbe7e8dd1","Type":"ContainerStarted","Data":"ad47461cb4d7b3738969afa6537beb3ebb32ca8fab1c6eeaa0e7a631c5143cfc"} Oct 31 19:17:07.435666 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:07.435627 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7lnww" event={"ID":"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe","Type":"ContainerStarted","Data":"dc133bf21a6930be5c3106f218d7e8417c194d06c78e0475cb83b2e2519489dd"} Oct 31 19:17:07.436155 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:07.435728 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:17:07.439777 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:07.439748 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerStarted","Data":"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc"} Oct 31 19:17:07.439911 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:07.439815 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerStarted","Data":"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054"} Oct 31 19:17:07.439911 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:07.439856 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerStarted","Data":"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c"} Oct 31 19:17:07.440002 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:07.439939 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerStarted","Data":"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3"} Oct 31 19:17:07.547877 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:07.547819 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7lnww" podStartSLOduration=67.346916363 podStartE2EDuration="1m11.547804645s" totalImagesPullingTime="4.200888282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:07.546776465 +0000 UTC m=+72.095446616" watchObservedRunningTime="2026-10-31 19:17:07.547804645 +0000 UTC m=+72.096474791" Oct 31 19:17:07.548250 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:07.548220 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f72jl" podStartSLOduration=69.893526633 podStartE2EDuration="1m11.548213572s" totalImagesPullingTime="1.654686939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:04.535038358 +0000 UTC m=+69.083708509" watchObservedRunningTime="2026-10-31 19:17:07.548213572 +0000 UTC m=+72.096883721" Oct 31 19:17:07.716496 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:07.716408 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.8446892630000002 podStartE2EDuration="10.716393787s" totalImagesPullingTime="6.855831594s" totalInitContainerRuntime="15.87293ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:07.716090572 +0000 UTC m=+72.264760736" watchObservedRunningTime="2026-10-31 19:17:07.716393787 +0000 UTC m=+72.265063934" Oct 31 19:17:08.294091 ip-10-0-128-164 kubenswrapper[2578]: I1031 19:17:08.294056 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:08.300587 ip-10-0-128-164 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 19:17:08.316363 ip-10-0-128-164 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 19:17:08.316608 ip-10-0-128-164 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 19:17:08.316677 ip-10-0-128-164 systemd[1]: kubelet.service: Consumed 3.978s CPU time. Oct 31 19:17:08.335752 ip-10-0-128-164 systemd[1]: Starting Kubernetes Kubelet... Oct 31 19:17:08.399310 ip-10-0-128-164 kubenswrapper[6107]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:17:08.399310 ip-10-0-128-164 kubenswrapper[6107]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 19:17:08.399310 ip-10-0-128-164 kubenswrapper[6107]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:17:08.399310 ip-10-0-128-164 kubenswrapper[6107]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:17:08.401719 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401599 6107 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.401719 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401714 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.401719 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401719 6107 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.401719 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401724 6107 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.401719 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401728 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401732 6107 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401736 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401740 6107 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401743 6107 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401747 6107 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401751 6107 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401755 6107 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401759 6107 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401763 6107 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401767 6107 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401771 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401775 6107 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401779 6107 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401783 6107 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401786 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401790 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401794 6107 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401798 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401802 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.402063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401806 6107 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401817 6107 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401822 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401826 6107 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401830 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401834 6107 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401838 6107 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401844 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401848 6107 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401852 6107 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401856 6107 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401860 6107 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401865 6107 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401869 6107 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401874 6107 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401896 6107 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401901 6107 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401905 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401912 6107 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.403014 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401927 6107 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401932 6107 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401936 6107 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401941 6107 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401945 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401949 6107 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401956 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401960 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401964 6107 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401969 6107 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401981 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401986 6107 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.401993 6107 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402000 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402006 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402011 6107 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402015 6107 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402019 6107 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402024 6107 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402028 6107 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.403798 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402033 6107 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402037 6107 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402041 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402046 6107 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402050 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402054 6107 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402058 6107 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402062 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402066 6107 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402070 6107 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402075 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402079 6107 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402084 6107 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402099 6107 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402104 6107 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402108 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402112 6107 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402116 6107 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402121 6107 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402125 6107 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.404674 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402130 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402134 6107 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402139 6107 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402144 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402147 6107 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402154 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402158 6107 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402162 6107 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402166 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402171 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402175 6107 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402179 6107 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402184 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402188 6107 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402193 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402197 6107 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402201 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402205 6107 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402209 6107 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.405547 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402214 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402218 6107 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402222 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.402227 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403585 6107 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403598 6107 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403603 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403609 6107 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403614 6107 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403618 6107 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403622 6107 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403626 6107 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403630 6107 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403635 6107 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403639 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403643 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403647 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403652 6107 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403656 6107 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403660 6107 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.406454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403664 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403668 6107 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403673 6107 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403677 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403681 6107 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403686 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403690 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403695 6107 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403699 6107 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403703 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403707 6107 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403711 6107 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403716 6107 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403720 6107 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403727 6107 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403736 6107 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403742 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403746 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403751 6107 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.407385 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403755 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403760 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403764 6107 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403769 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403773 6107 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403777 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403781 6107 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403786 6107 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403790 6107 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403794 6107 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403798 6107 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403801 6107 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403805 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403809 6107 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403813 6107 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403816 6107 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403820 6107 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403824 6107 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403829 6107 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403833 6107 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.408166 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403838 6107 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403842 6107 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403847 6107 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403851 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403855 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403859 6107 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403863 6107 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403867 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403874 6107 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403895 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403899 6107 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403904 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403910 6107 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403914 6107 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403919 6107 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403924 6107 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403928 6107 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403932 6107 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403936 6107 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403940 6107 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.408688 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403944 6107 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403949 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403953 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403957 6107 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403961 6107 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403965 6107 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403969 6107 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403973 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403978 6107 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403982 6107 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403988 6107 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.403995 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404000 6107 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404004 6107 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404008 6107 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404012 6107 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404017 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404021 6107 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404025 6107 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.409195 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404029 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404035 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404039 6107 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404044 6107 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404048 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404052 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404056 6107 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404060 6107 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404066 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404071 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404075 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404079 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404868 6107 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404894 6107 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404900 6107 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404904 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404908 6107 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404913 6107 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404917 6107 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.409818 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404921 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404925 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404930 6107 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404935 6107 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404940 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404944 6107 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404949 6107 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404953 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404957 6107 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404961 6107 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404966 6107 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404970 6107 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404974 6107 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404978 6107 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404985 6107 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404989 6107 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.404993 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405000 6107 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405006 6107 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.410349 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405011 6107 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405015 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405020 6107 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405025 6107 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405030 6107 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405034 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405039 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405043 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405047 6107 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405051 6107 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405055 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405059 6107 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405063 6107 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405068 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405072 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405076 6107 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405080 6107 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405085 6107 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405089 6107 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405093 6107 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.410813 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405097 6107 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405101 6107 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405106 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405110 6107 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405114 6107 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405118 6107 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405122 6107 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405128 6107 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405132 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405137 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405140 6107 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405145 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405149 6107 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405154 6107 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405158 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405163 6107 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405167 6107 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405172 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405176 6107 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.411346 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405180 6107 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405184 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405188 6107 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405193 6107 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405197 6107 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405203 6107 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405207 6107 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405211 6107 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405215 6107 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405220 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405224 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405229 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405233 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405238 6107 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405242 6107 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405246 6107 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405251 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405255 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405259 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.411777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405263 6107 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405270 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405275 6107 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405279 6107 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405284 6107 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405287 6107 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405292 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405296 6107 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405300 6107 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405308 6107 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405315 6107 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405320 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405325 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405329 6107 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405333 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405337 6107 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405342 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405346 6107 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405350 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.412274 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405354 6107 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405358 6107 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.405363 6107 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405508 6107 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405524 6107 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405534 6107 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405541 6107 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405549 6107 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405554 6107 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405560 6107 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405567 6107 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405572 6107 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405584 6107 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405590 6107 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405597 6107 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405602 6107 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405607 6107 flags.go:64] FLAG: --cgroup-root="" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405612 6107 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405635 6107 flags.go:64] FLAG: --client-ca-file="" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405639 6107 flags.go:64] FLAG: --cloud-provider="external" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405644 6107 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405652 6107 flags.go:64] FLAG: --cluster-domain="" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405657 6107 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 19:17:08.412726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405663 6107 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405676 6107 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405682 6107 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405689 6107 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405694 6107 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405699 6107 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405705 6107 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405710 6107 flags.go:64] FLAG: --contention-profiling="false" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405714 6107 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405719 6107 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405725 6107 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405729 6107 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405736 6107 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405742 6107 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405747 6107 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405752 6107 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405757 6107 flags.go:64] FLAG: --enable-server="true" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405762 6107 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405772 6107 flags.go:64] FLAG: --event-burst="100" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405777 6107 flags.go:64] FLAG: --event-qps="50" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405782 6107 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405787 6107 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405793 6107 flags.go:64] FLAG: --eviction-hard="" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405801 6107 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 19:17:08.413294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405806 6107 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405811 6107 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405816 6107 flags.go:64] FLAG: --eviction-soft="" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405821 6107 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405826 6107 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405831 6107 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405836 6107 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405841 6107 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405846 6107 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405851 6107 flags.go:64] FLAG: --feature-gates="" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405867 6107 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405873 6107 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405893 6107 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405899 6107 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405907 6107 flags.go:64] FLAG: --healthz-port="10248" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405912 6107 flags.go:64] FLAG: --help="false" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405917 6107 flags.go:64] FLAG: --hostname-override="ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405923 6107 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405928 6107 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405933 6107 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405938 6107 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405944 6107 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405949 6107 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 19:17:08.413980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405954 6107 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405958 6107 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405963 6107 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405968 6107 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405973 6107 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405978 6107 flags.go:64] FLAG: --kube-reserved="" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405984 6107 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405989 6107 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.405996 6107 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406000 6107 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406005 6107 flags.go:64] FLAG: --lock-file="" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406010 6107 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406015 6107 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406020 6107 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406030 6107 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406035 6107 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406040 6107 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406045 6107 flags.go:64] FLAG: --logging-format="text" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406049 6107 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406055 6107 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406060 6107 flags.go:64] FLAG: --manifest-url="" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406065 6107 flags.go:64] FLAG: --manifest-url-header="" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406073 6107 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406080 6107 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406087 6107 flags.go:64] FLAG: --max-pods="110" Oct 31 19:17:08.414911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406092 6107 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406097 6107 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406101 6107 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406106 6107 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406111 6107 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406116 6107 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406121 6107 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406133 6107 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406138 6107 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406143 6107 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406147 6107 flags.go:64] FLAG: --pod-cidr="" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406152 6107 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406157 6107 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406162 6107 flags.go:64] FLAG: --pods-per-core="0" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406167 6107 flags.go:64] FLAG: --port="10250" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406174 6107 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406179 6107 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0dfaf2352953a1145" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406184 6107 flags.go:64] FLAG: --qos-reserved="" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406189 6107 flags.go:64] FLAG: --read-only-port="10255" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406193 6107 flags.go:64] FLAG: --register-node="true" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406198 6107 flags.go:64] FLAG: --register-with-taints="" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406205 6107 flags.go:64] FLAG: --registry-burst="10" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406209 6107 flags.go:64] FLAG: --registry-qps="5" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406214 6107 flags.go:64] FLAG: --reserved-cpus="" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406219 6107 flags.go:64] FLAG: --reserved-memory="" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406225 6107 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 19:17:08.415536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406230 6107 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406235 6107 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406241 6107 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406245 6107 flags.go:64] FLAG: --runonce="false" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406252 6107 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406258 6107 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406263 6107 flags.go:64] FLAG: --seccomp-default="false" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406268 6107 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406273 6107 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406278 6107 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406283 6107 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406288 6107 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406293 6107 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406298 6107 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406302 6107 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406307 6107 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406312 6107 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406318 6107 flags.go:64] FLAG: --system-cgroups="" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406322 6107 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406331 6107 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406336 6107 flags.go:64] FLAG: --tls-cert-file="" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406343 6107 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406354 6107 flags.go:64] FLAG: --tls-min-version="" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406359 6107 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406364 6107 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 19:17:08.416166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406369 6107 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406373 6107 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406378 6107 flags.go:64] FLAG: --v="2" Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406385 6107 flags.go:64] FLAG: --version="false" Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406393 6107 flags.go:64] FLAG: --vmodule="" Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406399 6107 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.406404 6107 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406613 6107 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406620 6107 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406624 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406630 6107 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406636 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406641 6107 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406645 6107 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406649 6107 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406654 6107 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406658 6107 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406662 6107 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406668 6107 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406672 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.416756 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406676 6107 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406683 6107 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406689 6107 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406695 6107 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406702 6107 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406708 6107 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406713 6107 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406719 6107 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406724 6107 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406729 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406733 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406737 6107 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406741 6107 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406745 6107 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406750 6107 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406754 6107 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406758 6107 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406763 6107 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406768 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.417253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406772 6107 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406776 6107 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406780 6107 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406785 6107 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406792 6107 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406796 6107 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406801 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406805 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406809 6107 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406813 6107 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406818 6107 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406823 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406827 6107 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406832 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406836 6107 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406853 6107 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406857 6107 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406862 6107 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406866 6107 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406870 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.417804 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406874 6107 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406896 6107 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406900 6107 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406905 6107 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406909 6107 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406913 6107 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406917 6107 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406920 6107 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406925 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406929 6107 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406933 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406937 6107 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406942 6107 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406946 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406950 6107 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406954 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406964 6107 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406970 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406974 6107 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406979 6107 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.418321 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406983 6107 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406987 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406992 6107 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.406996 6107 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407001 6107 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407005 6107 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407010 6107 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407015 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407019 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407023 6107 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407027 6107 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407032 6107 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407036 6107 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407040 6107 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407044 6107 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407048 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407053 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407057 6107 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407061 6107 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407065 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.418800 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407069 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407074 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407078 6107 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407082 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407087 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407091 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407095 6107 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407099 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407105 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407110 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407114 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407119 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407123 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407127 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.407170 6107 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407233 6107 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.419358 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407240 6107 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407245 6107 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407251 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407255 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407260 6107 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407264 6107 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407269 6107 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407273 6107 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407277 6107 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407281 6107 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407286 6107 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407290 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407295 6107 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407301 6107 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407305 6107 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407309 6107 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407314 6107 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407318 6107 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407322 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407326 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.419777 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407330 6107 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407334 6107 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407338 6107 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407343 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407349 6107 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407354 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407358 6107 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407363 6107 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407367 6107 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407372 6107 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407376 6107 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407380 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407384 6107 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407388 6107 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407392 6107 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407396 6107 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407400 6107 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407403 6107 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407408 6107 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407412 6107 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.420290 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407416 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407421 6107 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407425 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407429 6107 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407434 6107 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407440 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407444 6107 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407449 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407453 6107 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407457 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407462 6107 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407467 6107 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407474 6107 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407478 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407483 6107 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407487 6107 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407493 6107 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407498 6107 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407502 6107 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407506 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.420904 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407511 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407515 6107 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407520 6107 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407524 6107 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407528 6107 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407532 6107 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407536 6107 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407540 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407545 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407549 6107 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407553 6107 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407557 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407561 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407565 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407570 6107 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407574 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407578 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407586 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407590 6107 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407594 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.421633 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407598 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407603 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407607 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407611 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407615 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407619 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407623 6107 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407627 6107 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407633 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407637 6107 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407641 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407646 6107 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407650 6107 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407654 6107 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407659 6107 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407663 6107 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407668 6107 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407672 6107 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407676 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.422138 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407680 6107 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407686 6107 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407692 6107 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407700 6107 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407707 6107 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.407712 6107 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.411578 6107 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.411596 6107 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.411614 6107 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.411619 6107 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411673 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411678 6107 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411681 6107 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411684 6107 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411687 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411690 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411692 6107 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411695 6107 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411698 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.422588 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411700 6107 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411703 6107 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411705 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411709 6107 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411711 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411715 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411718 6107 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411721 6107 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411724 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411726 6107 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411729 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411732 6107 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411735 6107 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411737 6107 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411740 6107 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411743 6107 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411745 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411748 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411750 6107 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.423055 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411753 6107 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411756 6107 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411759 6107 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411762 6107 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411765 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411767 6107 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411770 6107 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411773 6107 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411775 6107 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411780 6107 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411784 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411787 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411790 6107 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411793 6107 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411795 6107 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411798 6107 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411800 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411803 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411806 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.423489 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411809 6107 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411811 6107 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411814 6107 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411817 6107 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411820 6107 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411822 6107 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411825 6107 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411828 6107 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411832 6107 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411835 6107 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411838 6107 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411840 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411842 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411845 6107 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411847 6107 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411855 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411858 6107 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411861 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411863 6107 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411866 6107 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.423995 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411868 6107 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411871 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411873 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411876 6107 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411892 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411896 6107 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411898 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411901 6107 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411904 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411906 6107 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411909 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411912 6107 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411915 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411917 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411920 6107 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411922 6107 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411925 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411928 6107 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411930 6107 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.424648 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411932 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411935 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411937 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411940 6107 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411942 6107 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411945 6107 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411948 6107 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411950 6107 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411953 6107 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411956 6107 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411959 6107 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411963 6107 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411965 6107 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411968 6107 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411971 6107 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411974 6107 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411976 6107 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411979 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411982 6107 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.425341 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.411984 6107 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.412012 6107 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412166 6107 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412170 6107 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412173 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412176 6107 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412179 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412181 6107 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412184 6107 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412186 6107 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412189 6107 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412191 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412194 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412197 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412199 6107 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412202 6107 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.425981 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412205 6107 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412208 6107 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412210 6107 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412214 6107 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412218 6107 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412221 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412223 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412232 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412234 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412237 6107 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412240 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412242 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412245 6107 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412247 6107 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412250 6107 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412252 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412255 6107 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412257 6107 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412260 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412262 6107 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.426501 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412265 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412267 6107 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412270 6107 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412272 6107 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412275 6107 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412277 6107 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412281 6107 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412284 6107 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412287 6107 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412290 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412293 6107 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412295 6107 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412298 6107 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412301 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412303 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412306 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412308 6107 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412311 6107 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412313 6107 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.427134 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412315 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412324 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412326 6107 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412329 6107 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412331 6107 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412334 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412337 6107 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412339 6107 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412341 6107 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412344 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412346 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412349 6107 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412352 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412354 6107 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412357 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412359 6107 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412362 6107 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412364 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412367 6107 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412369 6107 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.427577 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412371 6107 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412374 6107 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412382 6107 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412384 6107 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412387 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412390 6107 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412392 6107 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412395 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412397 6107 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412400 6107 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412402 6107 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412405 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412407 6107 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412410 6107 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412418 6107 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412421 6107 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412423 6107 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412425 6107 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412428 6107 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.428062 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412430 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412433 6107 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412435 6107 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412438 6107 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412440 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412443 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412445 6107 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412448 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412450 6107 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412453 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412455 6107 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412458 6107 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412460 6107 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412462 6107 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.412485 6107 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412521 6107 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.428492 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412525 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412528 6107 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412530 6107 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412533 6107 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412535 6107 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412538 6107 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412541 6107 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412543 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412546 6107 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412548 6107 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412551 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412554 6107 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412562 6107 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412565 6107 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412567 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412570 6107 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412572 6107 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412575 6107 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412577 6107 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412579 6107 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.428867 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412582 6107 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412584 6107 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412589 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412592 6107 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412595 6107 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412598 6107 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412600 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412603 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412605 6107 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412608 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412610 6107 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412612 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412615 6107 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412617 6107 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412620 6107 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412622 6107 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412626 6107 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412629 6107 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412632 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.429354 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412634 6107 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412637 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412639 6107 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412642 6107 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412644 6107 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412647 6107 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412655 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412658 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412660 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412663 6107 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412665 6107 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412668 6107 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412671 6107 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412673 6107 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412677 6107 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412682 6107 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412685 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412688 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412691 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412694 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.429864 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412697 6107 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412699 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412702 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412704 6107 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412706 6107 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412709 6107 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412712 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412714 6107 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412717 6107 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412720 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412723 6107 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412725 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412728 6107 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412731 6107 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412733 6107 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412736 6107 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412739 6107 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412742 6107 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412746 6107 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.430355 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412754 6107 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412757 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412761 6107 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412764 6107 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412767 6107 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412770 6107 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412772 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412775 6107 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412778 6107 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412780 6107 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412783 6107 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412785 6107 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412788 6107 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412790 6107 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412793 6107 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412795 6107 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412798 6107 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412800 6107 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412803 6107 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412805 6107 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.430952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412808 6107 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412810 6107 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412814 6107 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412818 6107 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412820 6107 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412823 6107 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:08.412825 6107 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.412908 6107 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.415367 6107 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.415438 6107 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.415641 6107 server.go:1014] "Starting client certificate rotation" Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.415788 6107 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.415832 6107 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 19:10:56 +0000 UTC" deadline="2028-06-23 09:48:02.584820614 +0000 UTC" Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.415867 6107 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14414h30m54.168956947s" Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.416146 6107 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.417197 6107 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:17:08.431810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.418451 6107 log.go:25] "Validated CRI v1 runtime API" Oct 31 19:17:08.432411 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.420468 6107 log.go:25] "Validated CRI v1 image API" Oct 31 19:17:08.432411 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.421209 6107 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 19:17:08.432411 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.423348 6107 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7e827645-b8f0-4bd4-8392-b8ef4d590083:/dev/nvme0n1p3 b8ed8738-120f-48db-a50f-a4e84c100ee2:/dev/nvme0n1p4] Oct 31 19:17:08.432522 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.423367 6107 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b83d6279ed781dee8360c7d1fb625aa60d82cab0349f98742c8d053afcb889c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b83d6279ed781dee8360c7d1fb625aa60d82cab0349f98742c8d053afcb889c/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/223beb27e8090e99901d90d5ea58cae2b9a5cf3f3de4eb899f4524ca9c8c2c02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/223beb27e8090e99901d90d5ea58cae2b9a5cf3f3de4eb899f4524ca9c8c2c02/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33d65251ad167927b3f30e95c3c44a82f4e1c248f792d13bcbb6a27ef5e904db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33d65251ad167927b3f30e95c3c44a82f4e1c248f792d13bcbb6a27ef5e904db/userdata/shm major:0 minor:381 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3647333a5633307897889f185158cebaf5d49c9c638f0b932a8cac97cf23439f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3647333a5633307897889f185158cebaf5d49c9c638f0b932a8cac97cf23439f/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39876f5809d577b25073f5356d4f81793dcbe817779f869fb428725a1fd6b68e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39876f5809d577b25073f5356d4f81793dcbe817779f869fb428725a1fd6b68e/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40f3088e96e3723af2d48668d39a9bc14af3f66038c105289ef1a00eaf7d60b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40f3088e96e3723af2d48668d39a9bc14af3f66038c105289ef1a00eaf7d60b8/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5124428798b63af291653d7e0cd2cb62ef3da0b924cdd015b46d27e445abf261/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5124428798b63af291653d7e0cd2cb62ef3da0b924cdd015b46d27e445abf261/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5735a738be1b3e5df12b00db7aaf9420ec75770bcb20600c3fde66fa606c165f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5735a738be1b3e5df12b00db7aaf9420ec75770bcb20600c3fde66fa606c165f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/595187ad7830077771218255120ad2563a91877595b00108b32f7f237fa734d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/595187ad7830077771218255120ad2563a91877595b00108b32f7f237fa734d2/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b92c0b3aba96f21982c7ad2c7aaf06d4f29788cc48c9f073e004531f51e3b04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b92c0b3aba96f21982c7ad2c7aaf06d4f29788cc48c9f073e004531f51e3b04/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6056b78227e51f31f0bb4e7b7c0cccb905810522bb003cec3fd9753ad5a9cf35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6056b78227e51f31f0bb4e7b7c0cccb905810522bb003cec3fd9753ad5a9cf35/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f4eb791676db0e46c79c668c61b8f65927d25d6a942bd9d7a49d5657b7e34c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f4eb791676db0e46c79c668c61b8f65927d25d6a942bd9d7a49d5657b7e34c9/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76d1cc3979571efddf33afe750f327b28d66c780f5e27b637f0873adfc6d2c6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76d1cc3979571efddf33afe750f327b28d66c780f5e27b637f0873adfc6d2c6f/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82fbb210bc636df8d5b25f0f20c6ea4598aecfe78a54daf7227152438d0386ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82fbb210bc636df8d5b25f0f20c6ea4598aecfe78a54daf7227152438d0386ce/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8fc52594fbc580ce4ed493bdf7ec0ab5245602e6c62f8c43cfad99ebd61a70b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8fc52594fbc580ce4ed493bdf7ec0ab5245602e6c62f8c43cfad99ebd61a70b3/userdata/shm major:0 minor:365 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95d91696c8a2a094ee217a9ec78b15c42fc68cdd354e4db542ed2ed08c638006/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95d91696c8a2a094ee217a9ec78b15c42fc68cdd354e4db542ed2ed08c638006/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b529eed16eeea08a460c5cbc2490192432094395ebf6824840bc6cf8fbd64c02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b529eed16eeea08a460c5cbc2490192432094395ebf6824840bc6cf8fbd64c02/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca8eeef73881173f6f6d5579bc69430c6f9eeda3671cde33734be427a5bec078/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca8eeef73881173f6f6d5579bc69430c6f9eeda3671cde33734be427a5bec078/userdata/shm major:0 minor:379 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d64471645d3a8476fc189257c351d44f1801c4de7558ebe4f9b8c740bbd3617e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d64471645d3a8476fc189257c351d44f1801c4de7558ebe4f9b8c740bbd3617e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa92ed98c3bfc125b180abc5645a89a148875f624a3dfcfc2c88060eef095df2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa92ed98c3bfc125b180abc5645a89a148875f624a3dfcfc2c88060eef095df2/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc92348b50a855925ee720198d8ab164f99f78c91b179b19eae08713e3876371/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc92348b50a855925ee720198d8ab164f99f78c91b179b19eae08713e3876371/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0875c4d7-7bfd-45ab-ae34-75e20e327605/volumes/kubernetes.io~projected/kube-api-access-ngplm:{mountpoint:/var/lib/kubelet/pods/0875c4d7-7bfd-45ab-ae34-75e20e327605/volumes/kubernetes.io~projected/kube-api-access-ngplm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volume-subpaths/web-config/alertmanager/9 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volume-subpaths/web-config/config-reloader/6 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~empty-dir/config-out major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~projected/kube-api-access-mtq5n:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~projected/kube-api-access-mtq5n major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~projected/tls-assets major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/config-volume major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/web-config major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d851015-b321-4b76-b09f-45c307ecfb37/volumes/kubernetes.io~projected/kube-api-access-7r6m7:{mountpoint:/var/lib/kubelet/pods/2d851015-b321-4b76-b09f-45c307ecfb37/volumes/kubernetes.io~projected/kube-api-access-7r6m7 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d851015-b321-4b76-b09f-45c307ecfb37/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/2d851015-b321-4b76-b09f-45c307ecfb37/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3de0cf07-8e8a-47c6-a382-bc56884867e6/volumes/kubernetes.io~projected/kube-api-access-f9k9l:{mountpoint:/var/lib/kubelet/pods/3de0cf07-8e8a-47c6-a382-bc56884867e6/volumes/kubernetes.io~projected/kube-api-access-f9k9l major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/474d47d1-f185-4127-a085-786c6ada462e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/474d47d1-f185-4127-a085-786c6ada462e/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6310b145-4c78-4f58-91d0-be7558e3b1cc/volumes/kubernetes.io~projected/kube-api-access-xr4k7:{mountpoint:/var/lib/kubelet/pods/6310b145-4c78-4f58-91d0-be7558e3b1cc/volumes/kubernetes.io~projected/kube-api-access-xr4k7 major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6310b145-4c78-4f58-91d0-be7558e3b1cc/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6310b145-4c78-4f58-91d0-be7558e3b1cc/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6310b145-4c78-4f58-91d0-be7558e3b1cc/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/6310b145-4c78-4f58-91d0-be7558e3b1cc/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe/volumes/kubernetes.io~projected/kube-api-access-vcwm6:{mountpoint:/var/lib/kubelet/pods/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe/volumes/kubernetes.io~projected/kube-api-access-vcwm6 major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d131de0-3a07-4d70-99e7-7aedb4648f31/volumes/kubernetes.io~projected/kube-api-access-pqsxp:{mountpoint:/var/lib/kubelet/pods/6d131de0-3a07-4d70-99e7-7aedb4648f31/volumes/kubernetes.io~projected/kube-api-access-pqsxp major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d131de0-3a07-4d70-99e7-7aedb4648f31/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6d131de0-3a07-4d70-99e7-7aedb4648f31/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d131de0-3a07-4d70-99e7-7aedb4648f31/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/6d131de0-3a07-4d70-99e7-7aedb4648f31/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82d520a7-fab4-46a7-bed2-9d4b3291d198/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/82d520a7-fab4-46a7-bed2-9d4b3291d198/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82d520a7-fab4-46a7-bed2-9d4b3291d198/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/82d520a7-fab4-46a7-bed2-9d4b3291d198/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82d520a7-fab4-46a7-bed2-9d4b3291d198/volumes/kubernetes.io~projected/kube-api-access-d59s2:{mountpoint:/var/lib/kubelet/pods/82d520a7-fab4-46a7-bed2-9d4b3291d198/volumes/kubernetes.io~projected/kube-api-access-d59s2 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/890a45ca-fe6f-4c91-9fda-034924575914/volumes/kubernetes.io~projected/kube-api-access-62dqg:{mountpoint:/var/lib/kubelet/pods/890a45ca-fe6f-4c91-9fda-034924575914/volumes/kubernetes.io~projected/kube-api-access-62dqg major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d3cf005-2850-4952-b7ed-7aeeb299603e/volumes/kubernetes.io~projected/kube-api-access-bpck4:{mountpoint:/var/lib/kubelet/pods/8d3cf005-2850-4952-b7ed-7aeeb299603e/volumes/kubernetes.io~projected/kube-api-access-bpck4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volume-subpaths/web-config/config-reloader/2 major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volume-subpaths/web-config/prometheus/15 major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~empty-dir/config-out major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~projected/kube-api-access-vj4gh:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~projected/kube-api-access-vj4gh major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~projected/tls-assets major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/config major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/web-config major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f/volumes/kubernetes.io~projected/kube-api-access-zlppw:{mountpoint:/var/lib/kubelet/pods/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f/volumes/kubernetes.io~projected/kube-api-access-zlppw major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c680bdb-3c36-4584-b1ec-78ab49d07aa9/volumes/kubernetes.io~projected/kube-api-access-z96kc:{mountpoint:/var/lib/kubelet/pods/9c680bdb-3c36-4584-b1ec-78ab49d07aa9/volumes/kubernetes.io~projected/kube-api-access-z96kc major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c680bdb-3c36-4584-b1ec-78ab49d07aa9/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/9c680bdb-3c36-4584-b1ec-78ab49d07aa9/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a302be0c-ed10-4c2d-8673-debd526d4994/volumes/kubernetes.io~projected/kube-api-access-gtwl2:{mountpoint:/var/lib/kubelet/pods/a302be0c-ed10-4c2d-8673-debd526d4994/volumes/kubernetes.io~projected/kube-api-access-gtwl2 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a302be0c-ed10-4c2d-8673-debd526d4994/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a302be0c-ed10-4c2d-8673-debd526d4994/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b056a7-e321-4ce7-ab1e-0d2e2824869f/volumes/kubernetes.io~projected/kube-api-access-pw4wt:{mountpoint:/var/lib/kubelet/pods/c8b056a7-e321-4ce7-ab1e-0d2e2824869f/volumes/kubernetes.io~projected/kube-api-access-pw4wt major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d27497f4-b29e-46c2-8f63-664fbe7e8dd1/volumes/kubernetes.io~projected/kube-api-access-6jz44:{mountpoint:/var/lib/kubelet/pods/d27497f4-b29e-46c2-8f63-664fbe7e8dd1/volumes/kubernetes.io~projected/kube-api-access-6jz44 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d27497f4-b29e-46c2-8f63-664fbe7e8dd1/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/d27497f4-b29e-46c2-8f63-664fbe7e8dd1/volumes/kubernetes.io~secret/metrics-certs major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1441c08-c76a-4954-80c9-8c1a0a04ce9f/volumes/kubernetes.io~projected/kube-api-access-rps8q:{mountpoint:/var/lib/kubelet/pods/e1441c08-c76a-4954-80c9-8c1a0a04ce9f/volumes/kubernetes.io~projected/kube-api-access-rps8q major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1441c08-c76a-4954-80c9-8c1a0a04ce9f/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e1441c08-c76a-4954-80c9-8c1a0a04ce9f/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1441c08-c76a-4954-80c9-8c1a0a04ce9f/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/e1441c08-c76a-4954-80c9-8c1a0a04ce9f/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6c9d77e-0f26-4a74-a08b-a2aff00389d2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f6c9d77e-0f26-4a74-a08b-a2aff00389d2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6c9d77e-0f26-4a74-a08b-a2aff00389d2/volumes/kubernetes.io~projected/kube-api-access-x2d7k:{mountpoint:/var/lib/kubelet/pods/f6c9d77e-0f26-4a74-a08b-a2aff00389d2/volumes/kubernetes.io~projected/kube-api-access-x2d7k major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6c9d77e-0f26-4a74-a08b-a2aff00389d2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f6c9d77e-0f26-4a74-a08b-a2aff00389d2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f947add9-2413-4c26-860a-a5431e7cd2b9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f947add9-2413-4c26-860a-a5431e7cd2b9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:374 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1249690ed43fd9daf7586cd78545944143d7f65ea48bc7cf54db43a60f428adf/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e8ef78ffc44b019c237bb7b9250addf4096f6a40126fd82ebb710912d4703bf3/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8cc233f9ca91f50a286cfe71219fe75519117ee9005d9cc2b6c833470976f784/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/df28ef4e51e89ca948edcca1288aa524d0120b632137c4124bbbf1c3651e443e/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/92b7b701e89f429d7ff62e2c2d9063caaa01b96ca6ae9bfffd05964fba49f5b1/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7181a3c6e6e5fafa5167de19d29a14320dee42e7723c00f808f4881949a12bf6/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/157cdb46fb6af374db948004c8a1415e3174d78c5eab58cbd64414a0e890b7f8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f1bb3269b665944a269eb24538031e736f49a0aa44f839b0e762b66218cf845b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/900816677aadcfbb712fde45293ad91be0df67ef32b4901a367a43cddd14fd71/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/95aa29bd358786662ebbccbf4953d33136373311b64214ea08fc3fa55f74d9dc/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/8766ab6703ba61cf9511dcc47a061aa66769b39b6d3936f140b333e1cdf1cb33/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/39853834dd92584a65757d0ea2b2738303a4dffe0521e918df83106c1ec4c5b8/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/09a960c430da3710ddc17e690378a3df0cf87cacab4b6d870c3b21a32c4f7bc4/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/272f1ab57d5b730f72fec74ca96545230d63dd23d111b0c946ccbe75c6323ab6/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/000e17c23606ec7487d231ee59caecf8c7ef69bf7b8f0468923873f11e1b85b1/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/20a17aa472e085fd8dace4df9744f4e7a1f61fc737dc21d4b4f784a9610492d1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2a9ea04b487d0ad00e9c4f1b91e816d34284805135230a727afcd7a070d6ba21/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4d4667fbdcfa84472e1df23b40820fc6f3e189591198367794fb4fcd11c056aa/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ccf91ef4de795688b4211a41061953beae36260ddcbddab42e3a950a2143fad0/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e308ba1bac00876be485d34bda6cccd8a4b34fcd8315ceae0cda274c29399e26/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/fd551f3c51d7d627823886286ba480764c776d8bd94b991d086cb1cf016ac878/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7d6de7193bef99e1bcfa674555ee767b638e8b3dfa3e2060209ec0ebd5ce3b55/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/9c3070c35f871d3d757362c1567527513dbbb470dc310f11a7a100b3dfcce59b/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/e6b0b99d29080910e11515a1ff70464dd1cb3f5faf5ae21cc5b2d3661c724b4b/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/631494e8942518cc5090e6f0ba104659a61dcae36cef5e9fb76f5587f0769623/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/5c5e5bb1f538d4d58a1512798c1c0352fa6826b5420cb64ee7e1c381f149b0ac/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/e5498c7d721529de8df8d7d6830ab6d2d38d31c2812a88bc1ada15dade15f9af/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/77fcfb56dbe90db5206cfd47ca86d3c445404a420f6ac312854558ff315e2fb8/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/f806ec0b8cfa63033c3ed9bb7ac3e0be49185f9ed60c9d173e8a4ede3fe82340/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/9543127a3da4a927e68da13375dac092153d4f5a1ff832d323260d54957cb8a9/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/c25db14542b1069b74d2c185ef598b30e2f1c4b0c46df3b66954b11380f48c7a/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/8458fe01b451057c16daa310c871cfad9afb397e06997785a806f65a9a2935cb/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/74c5feeff380d1a12b75d751d5d74e280e7b0d2e98f788b228f2507242d138dc/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/8ec9a13ef08fe113bad0ef76587c4beaa75c7b8119ff560a2ed3ac0e1bbb0cc4/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/ac758d527ca938ee329776bd5a23ad6b78e2b9482ce125dd348179efd962e6f6/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/4bf87202398d100db78e8886e4e4a6806ace1532fbddad455181d2d60f04f573/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/1f5863d0050497b632b45fa99ba4fe60614a100371a9eaf52bbb1238a8a6e3df/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/a9f436edf2b7468deba1d009ac10b0f42b3a76de4f0e05a571d0021ff7b0bfa9/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/7ca16f24d16f9edd1da0fca7f34b91e0f9aea660fa37a6734e4068aa726b4ed8/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/53738fc8dfe81f6fa25baea287902fc16e8c315d7c3bb84f0a602e078ad3045f/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/9f7cf677803b5e8cb343d60f459be942198aa157d5d3dd5c0bbae6e6654e91cf/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/1b063ddf6bf5d1dfc6c53fe5d716bbeb39a974fc40795d98307a9aaf735e548d/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/bd9deeb9faf732772af061ab695834aa89f7c41fd09e50fc8c45abee9a372248/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/f50c2ebf8d251ddff6d8c4e571f541ad2cff869641cb310c72ebd22ebef1d43f/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/7c06bd66e21fd3f433b1294982856d32b7e055944b3feb9dd76ae6f5c5c551b8/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/84397ec572c989f8f0fa6507c98cd6d6937268fb3e530880d6a65c0590b377fe/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/ca6e103d1e3921b21d7ee4eaa87d7e3e3f93fefe6689f8867e73b53cd7736602/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/2d67576d1875510f56953ad3c3acbf61f824645f5793439f5e54e732dde5c597/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/f17159b0effa772675f0665f2f2efc29d4215f4d969a1a25e9efd6a8cff7ba79/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/ec0b2c2eccd4cba18a8f26da44b6381166926b29a4d5ba4097bed5fe1c638c81/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/6fdf0320f21d6df023fc93bf7ebdf6bb4c5174efb44e234f406faafa48440df2/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/3b7828709f435d337ab31beac6ba2e8a06782dc878b3c74a2119e65643903a43/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/1d3d986774135164dac1fa7d31dabb375bbce8122f53b0e3fdfdb56593c5ca60/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/f97c2acb90b97b695ab9055a3a776b73de99b50e9d3be0bb4dd03a3d12988c29/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/36cb13c4e4c7080a59c80928131e49e2dd0de87eff1d7ce1bc3cb1bf15586e0c/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/edf0443acdd9e858e1e657ba8ae6304526d9e10b22b0c75106ce054dcc90deb3/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/3f29259c34666b80f62fce9383351b04bc45151a3bf0b3e8e1baee313ca2a7b5/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/191995a4a9f5d6582a71225fd4c102892273d4b398818a0f746a81430a9794a0/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/b4da2127b725cbb6571fac177043e8bfd39d7d4b858bb1391d339a910e2afa8e/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/59cca221b64f3051b01454e97af0880fd1fef230a042207afc9c6de82455e9a3/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/836dce42aba7bbd0447cf1d511427af586226d0847f7147eab86dc4c1e8eba22/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-433:{mountpoint:/var/lib/containers/storage/overlay/6127fd51cef2bf1f418a04f8001a68e5f863bff2c42d21a04201e2d1847cd8f1/merged major:0 minor:433 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/71218246c2e144d175d8fa0b2549a52d8114f98828668bfb46bca6397ae87204/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e3b283e8a1a5a5520448744c01b3cbbff7a938367007c6a78e772858a63f4c55/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0ae8cb53bfce05539f6d189f192368d3942e8148d4a0a94cf89e614f1bd6799d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/d4cacc7c40a9965dc4039a41cdf7a99e3ea7ac22f912c6aa5e03281928615f30/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/1acfac821b4db05cc0464d884e2e5590b6c55e7693c8dccd9a77cf17bc8455e4/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/5e89c9a104db30ffd81bea84f0f0af4bb55e5837f1b85f348fd8a35bc010fa94/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/893eedbb72911608a17749cbca9d2acc2924f6ca46feb63f5ab2bce520e24f32/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/02a1620f1fed59cfb39e7b759103bc5173f1fc8dffc691dbf6c1fbaba68b35bf/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/aa082ce012d5e37052a1b7f3ec108e5f75a9abab2463a17ff61121655fcbdf29/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9bcfad7287aaca526b40f6ca91eeed1af3d6d78bb959d933590cb1f8b02315cf/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 19:17:08.432826 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.432409 6107 manager.go:217] Machine: {Timestamp:2026-10-31 19:17:08.431496628 +0000 UTC m=+0.064890529 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103939 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aaf51cf45e047196186a5ab70eab0 SystemUUID:ec2aaf51-cf45-e047-1961-86a5ab70eab0 BootID:a5f8d881-8504-48f3-8245-e8aa8a525751 Filesystems:[{Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:350 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/82d520a7-fab4-46a7-bed2-9d4b3291d198/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a302be0c-ed10-4c2d-8673-debd526d4994/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c680bdb-3c36-4584-b1ec-78ab49d07aa9/volumes/kubernetes.io~projected/kube-api-access-z96kc DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6310b145-4c78-4f58-91d0-be7558e3b1cc/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6d131de0-3a07-4d70-99e7-7aedb4648f31/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82d520a7-fab4-46a7-bed2-9d4b3291d198/volumes/kubernetes.io~projected/kube-api-access-d59s2 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3de0cf07-8e8a-47c6-a382-bc56884867e6/volumes/kubernetes.io~projected/kube-api-access-f9k9l DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f/volumes/kubernetes.io~projected/kube-api-access-zlppw DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d131de0-3a07-4d70-99e7-7aedb4648f31/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:259 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:357 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d3cf005-2850-4952-b7ed-7aeeb299603e/volumes/kubernetes.io~projected/kube-api-access-bpck4 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c680bdb-3c36-4584-b1ec-78ab49d07aa9/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d851015-b321-4b76-b09f-45c307ecfb37/volumes/kubernetes.io~projected/kube-api-access-7r6m7 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1441c08-c76a-4954-80c9-8c1a0a04ce9f/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~projected/kube-api-access-vj4gh DeviceMajor:0 DeviceMinor:364 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6056b78227e51f31f0bb4e7b7c0cccb905810522bb003cec3fd9753ad5a9cf35/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5735a738be1b3e5df12b00db7aaf9420ec75770bcb20600c3fde66fa606c165f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6310b145-4c78-4f58-91d0-be7558e3b1cc/volumes/kubernetes.io~projected/kube-api-access-xr4k7 DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:359 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8fc52594fbc580ce4ed493bdf7ec0ab5245602e6c62f8c43cfad99ebd61a70b3/userdata/shm DeviceMajor:0 DeviceMinor:365 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40f3088e96e3723af2d48668d39a9bc14af3f66038c105289ef1a00eaf7d60b8/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:358 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39876f5809d577b25073f5356d4f81793dcbe817779f869fb428725a1fd6b68e/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33d65251ad167927b3f30e95c3c44a82f4e1c248f792d13bcbb6a27ef5e904db/userdata/shm DeviceMajor:0 DeviceMinor:381 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/890a45ca-fe6f-4c91-9fda-034924575914/volumes/kubernetes.io~projected/kube-api-access-62dqg DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/474d47d1-f185-4127-a085-786c6ada462e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d851015-b321-4b76-b09f-45c307ecfb37/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3647333a5633307897889f185158cebaf5d49c9c638f0b932a8cac97cf23439f/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:354 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6c9d77e-0f26-4a74-a08b-a2aff00389d2/volumes/kubernetes.io~projected/kube-api-access-x2d7k DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b92c0b3aba96f21982c7ad2c7aaf06d4f29788cc48c9f073e004531f51e3b04/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d27497f4-b29e-46c2-8f63-664fbe7e8dd1/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:375 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f947add9-2413-4c26-860a-a5431e7cd2b9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:374 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/82d520a7-fab4-46a7-bed2-9d4b3291d198/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a302be0c-ed10-4c2d-8673-debd526d4994/volumes/kubernetes.io~projected/kube-api-access-gtwl2 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e1441c08-c76a-4954-80c9-8c1a0a04ce9f/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe/volumes/kubernetes.io~projected/kube-api-access-vcwm6 DeviceMajor:0 DeviceMinor:378 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/595187ad7830077771218255120ad2563a91877595b00108b32f7f237fa734d2/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc92348b50a855925ee720198d8ab164f99f78c91b179b19eae08713e3876371/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b83d6279ed781dee8360c7d1fb625aa60d82cab0349f98742c8d053afcb889c/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f4eb791676db0e46c79c668c61b8f65927d25d6a942bd9d7a49d5657b7e34c9/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa92ed98c3bfc125b180abc5645a89a148875f624a3dfcfc2c88060eef095df2/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82fbb210bc636df8d5b25f0f20c6ea4598aecfe78a54daf7227152438d0386ce/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d131de0-3a07-4d70-99e7-7aedb4648f31/volumes/kubernetes.io~projected/kube-api-access-pqsxp DeviceMajor:0 DeviceMinor:260 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca8eeef73881173f6f6d5579bc69430c6f9eeda3671cde33734be427a5bec078/userdata/shm DeviceMajor:0 DeviceMinor:379 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5124428798b63af291653d7e0cd2cb62ef3da0b924cdd015b46d27e445abf261/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6c9d77e-0f26-4a74-a08b-a2aff00389d2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-433 DeviceMajor:0 DeviceMinor:433 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:292 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/d27497f4-b29e-46c2-8f63-664fbe7e8dd1/volumes/kubernetes.io~projected/kube-api-access-6jz44 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b056a7-e321-4ce7-ab1e-0d2e2824869f/volumes/kubernetes.io~projected/kube-api-access-pw4wt DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~projected/kube-api-access-mtq5n DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:362 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6310b145-4c78-4f58-91d0-be7558e3b1cc/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b529eed16eeea08a460c5cbc2490192432094395ebf6824840bc6cf8fbd64c02/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6c9d77e-0f26-4a74-a08b-a2aff00389d2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95d91696c8a2a094ee217a9ec78b15c42fc68cdd354e4db542ed2ed08c638006/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e1441c08-c76a-4954-80c9-8c1a0a04ce9f/volumes/kubernetes.io~projected/kube-api-access-rps8q DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76d1cc3979571efddf33afe750f327b28d66c780f5e27b637f0873adfc6d2c6f/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:363 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/0875c4d7-7bfd-45ab-ae34-75e20e327605/volumes/kubernetes.io~projected/kube-api-access-ngplm DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d64471645d3a8476fc189257c351d44f1801c4de7558ebe4f9b8c740bbd3617e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/223beb27e8090e99901d90d5ea58cae2b9a5cf3f3de4eb899f4524ca9c8c2c02/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1b83d6279ed781d MacAddress:ae:be:d9:17:ea:7d Speed:10000 Mtu:8901} {Name:33d65251ad16792 MacAddress:3a:5e:f3:94:33:b5 Speed:10000 Mtu:8901} {Name:3647333a5633307 MacAddress:1a:20:1e:3b:ab:b7 Speed:10000 Mtu:8901} {Name:39876f5809d577b MacAddress:a2:5b:2a:84:c9:d9 Speed:10000 Mtu:8901} {Name:40f3088e96e3723 MacAddress:06:3f:ac:17:a9:61 Speed:10000 Mtu:8901} {Name:595187ad7830077 MacAddress:92:07:5e:f0:9e:2e Speed:10000 Mtu:8901} {Name:5b92c0b3aba96f2 MacAddress:1a:4d:b3:d6:9f:94 Speed:10000 Mtu:8901} {Name:8fc52594fbc580c MacAddress:b2:35:f2:67:dc:df Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f6:a2:2c:6f Speed:0 Mtu:9001} {Name:br-int MacAddress:a2:3e:9f:76:84:96 Speed:0 Mtu:8901} {Name:ca8eeef73881173 MacAddress:0e:6b:20:63:67:9b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f6:a2:2c:6f Speed:0 Mtu:9001} {Name:fc92348b50a8559 MacAddress:6a:32:86:26:bf:21 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:22:62:ff:45:67:d8 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d6:a9:71:a8:f4:66 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 19:17:08.433045 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.432830 6107 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 19:17:08.433045 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.432899 6107 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 19:17:08.433211 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433181 6107 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 19:17:08.433333 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433212 6107 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-164.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 19:17:08.433372 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433336 6107 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 19:17:08.433372 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433344 6107 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 19:17:08.433372 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433357 6107 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 19:17:08.433447 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433378 6107 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 19:17:08.433481 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433473 6107 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 19:17:08.433696 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433686 6107 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 19:17:08.433806 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433798 6107 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 19:17:08.433841 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433833 6107 kubelet.go:493] "Attempting to sync node with API server" Oct 31 19:17:08.433868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433849 6107 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 19:17:08.433868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433859 6107 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 19:17:08.433868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433868 6107 kubelet.go:405] "Adding apiserver pod source" Oct 31 19:17:08.434010 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.433876 6107 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 19:17:08.434618 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.434604 6107 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 19:17:08.434618 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.434617 6107 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 19:17:08.435089 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.435068 6107 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 19:17:08.437763 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.437748 6107 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 19:17:08.437763 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.437762 6107 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 19:17:08.437873 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.437853 6107 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 19:17:08.438061 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438049 6107 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 19:17:08.438093 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438079 6107 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 19:17:08.438184 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438174 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 19:17:08.438218 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438189 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 19:17:08.438218 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438199 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 19:17:08.438218 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438208 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 19:17:08.438218 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438214 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 19:17:08.438350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438220 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 19:17:08.438350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438226 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 19:17:08.438350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438231 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 19:17:08.438350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438238 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 19:17:08.438350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438244 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 19:17:08.438350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438258 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 19:17:08.438350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438269 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 19:17:08.438350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438301 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 19:17:08.438350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.438308 6107 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 19:17:08.441770 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.441753 6107 server.go:1263] "Started kubelet" Oct 31 19:17:08.441890 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.441801 6107 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 19:17:08.442345 ip-10-0-128-164 systemd[1]: Started Kubernetes Kubelet. Oct 31 19:17:08.442857 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.442306 6107 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 19:17:08.442857 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.442387 6107 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 19:17:08.444125 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.443044 6107 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 19:17:08.444125 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.443339 6107 server.go:317] "Adding debug handlers to kubelet server" Oct 31 19:17:08.447985 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.447741 6107 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 19:17:08.447985 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.447764 6107 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 19:17:08.447985 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.447819 6107 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:10:58 +0000 UTC" deadline="2028-06-21 18:05:49.190387681 +0000 UTC" Oct 31 19:17:08.447985 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.447837 6107 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14374h48m40.742553517s" Oct 31 19:17:08.447985 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:08.447865 6107 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:17:08.447985 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.447896 6107 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 19:17:08.447985 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.447908 6107 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 19:17:08.447985 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.447929 6107 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 19:17:08.450682 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.450660 6107 factory.go:55] Registering systemd factory Oct 31 19:17:08.450773 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.450686 6107 factory.go:223] Registration of the systemd container factory successfully Oct 31 19:17:08.451136 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.451119 6107 factory.go:153] Registering CRI-O factory Oct 31 19:17:08.451136 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.451137 6107 factory.go:223] Registration of the crio container factory successfully Oct 31 19:17:08.451294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.451202 6107 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 19:17:08.451294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.451225 6107 factory.go:103] Registering Raw factory Oct 31 19:17:08.451294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.451238 6107 manager.go:1199] Started watching for new ooms in manager Oct 31 19:17:08.451567 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.451553 6107 manager.go:319] Starting recovery of all containers Oct 31 19:17:08.454415 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454372 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-db" seLinuxMountContext="" Oct 31 19:17:08.454508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454418 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c680bdb-3c36-4584-b1ec-78ab49d07aa9" volumeName="kubernetes.io/projected/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-kube-api-access-z96kc" seLinuxMountContext="" Oct 31 19:17:08.454508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454435 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d851015-b321-4b76-b09f-45c307ecfb37" volumeName="kubernetes.io/configmap/2d851015-b321-4b76-b09f-45c307ecfb37-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 19:17:08.454508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454443 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3de0cf07-8e8a-47c6-a382-bc56884867e6" volumeName="kubernetes.io/configmap/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-daemon-config" seLinuxMountContext="" Oct 31 19:17:08.454508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454453 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d131de0-3a07-4d70-99e7-7aedb4648f31" volumeName="kubernetes.io/secret/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 19:17:08.454508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454464 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-metrics-client-ca" seLinuxMountContext="" Oct 31 19:17:08.454508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454476 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 19:17:08.454508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454490 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 31 19:17:08.454508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454508 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-kube-api-access-mtq5n" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454521 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d131de0-3a07-4d70-99e7-7aedb4648f31" volumeName="kubernetes.io/configmap/6d131de0-3a07-4d70-99e7-7aedb4648f31-metrics-client-ca" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454533 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d131de0-3a07-4d70-99e7-7aedb4648f31" volumeName="kubernetes.io/secret/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-tls" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454545 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890a45ca-fe6f-4c91-9fda-034924575914" volumeName="kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-cni-binary-copy" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454570 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-metrics-client-certs" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454583 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a302be0c-ed10-4c2d-8673-debd526d4994" volumeName="kubernetes.io/projected/a302be0c-ed10-4c2d-8673-debd526d4994-kube-api-access-gtwl2" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454599 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6c9d77e-0f26-4a74-a08b-a2aff00389d2" volumeName="kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovnkube-config" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454613 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64d8b12c-0fc1-4b11-9a11-8155f18c8dbe" volumeName="kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454628 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" volumeName="kubernetes.io/projected/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-kube-api-access-6jz44" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454652 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0875c4d7-7bfd-45ab-ae34-75e20e327605" volumeName="kubernetes.io/empty-dir/0875c4d7-7bfd-45ab-ae34-75e20e327605-tmp-dir" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454666 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454679 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d131de0-3a07-4d70-99e7-7aedb4648f31" volumeName="kubernetes.io/configmap/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454701 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c680bdb-3c36-4584-b1ec-78ab49d07aa9" volumeName="kubernetes.io/configmap/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-config-volume" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454714 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3de0cf07-8e8a-47c6-a382-bc56884867e6" volumeName="kubernetes.io/projected/3de0cf07-8e8a-47c6-a382-bc56884867e6-kube-api-access-f9k9l" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454729 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="474d47d1-f185-4127-a085-786c6ada462e" volumeName="kubernetes.io/secret/474d47d1-f185-4127-a085-786c6ada462e-agent-certs" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454741 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890a45ca-fe6f-4c91-9fda-034924575914" volumeName="kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454754 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-web-config" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454779 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454815 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454827 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1441c08-c76a-4954-80c9-8c1a0a04ce9f" volumeName="kubernetes.io/projected/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-api-access-rps8q" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454857 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6310b145-4c78-4f58-91d0-be7558e3b1cc" volumeName="kubernetes.io/configmap/6310b145-4c78-4f58-91d0-be7558e3b1cc-metrics-client-ca" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454869 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454926 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c680bdb-3c36-4584-b1ec-78ab49d07aa9" volumeName="kubernetes.io/secret/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-metrics-tls" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454937 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6c9d77e-0f26-4a74-a08b-a2aff00389d2" volumeName="kubernetes.io/secret/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454949 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f947add9-2413-4c26-860a-a5431e7cd2b9" volumeName="kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454961 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0875c4d7-7bfd-45ab-ae34-75e20e327605" volumeName="kubernetes.io/projected/0875c4d7-7bfd-45ab-ae34-75e20e327605-kube-api-access-ngplm" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454972 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3de0cf07-8e8a-47c6-a382-bc56884867e6" volumeName="kubernetes.io/configmap/3de0cf07-8e8a-47c6-a382-bc56884867e6-cni-binary-copy" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454982 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="474d47d1-f185-4127-a085-786c6ada462e" volumeName="kubernetes.io/configmap/474d47d1-f185-4127-a085-786c6ada462e-konnectivity-ca" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.454994 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455005 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d851015-b321-4b76-b09f-45c307ecfb37" volumeName="kubernetes.io/empty-dir/2d851015-b321-4b76-b09f-45c307ecfb37-data-volume" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455020 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890a45ca-fe6f-4c91-9fda-034924575914" volumeName="kubernetes.io/projected/890a45ca-fe6f-4c91-9fda-034924575914-kube-api-access-62dqg" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455031 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d3cf005-2850-4952-b7ed-7aeeb299603e" volumeName="kubernetes.io/projected/8d3cf005-2850-4952-b7ed-7aeeb299603e-kube-api-access-bpck4" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455042 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f" volumeName="kubernetes.io/projected/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-kube-api-access-zlppw" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455055 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1441c08-c76a-4954-80c9-8c1a0a04ce9f" volumeName="kubernetes.io/empty-dir/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-volume-directive-shadow" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455065 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1441c08-c76a-4954-80c9-8c1a0a04ce9f" volumeName="kubernetes.io/secret/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455078 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1441c08-c76a-4954-80c9-8c1a0a04ce9f" volumeName="kubernetes.io/secret/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455091 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-metrics-client-ca" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455103 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d851015-b321-4b76-b09f-45c307ecfb37" volumeName="kubernetes.io/projected/2d851015-b321-4b76-b09f-45c307ecfb37-kube-api-access-7r6m7" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455114 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-config-out" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455126 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455137 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-config-out" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455150 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-tls-assets" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455162 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455173 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455185 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-tls-assets" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455197 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-kube-api-access-vj4gh" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455208 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d27497f4-b29e-46c2-8f63-664fbe7e8dd1" volumeName="kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455220 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1441c08-c76a-4954-80c9-8c1a0a04ce9f" volumeName="kubernetes.io/configmap/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455252 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-main-db" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455263 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455277 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6310b145-4c78-4f58-91d0-be7558e3b1cc" volumeName="kubernetes.io/secret/6310b145-4c78-4f58-91d0-be7558e3b1cc-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455288 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455300 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-config" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455311 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-grpc-tls" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455324 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6c9d77e-0f26-4a74-a08b-a2aff00389d2" volumeName="kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-env-overrides" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455338 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6c9d77e-0f26-4a74-a08b-a2aff00389d2" volumeName="kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovnkube-script-lib" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455349 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-cluster-tls-config" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455362 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-web-config" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455372 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455384 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d131de0-3a07-4d70-99e7-7aedb4648f31" volumeName="kubernetes.io/empty-dir/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-textfile" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455401 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82d520a7-fab4-46a7-bed2-9d4b3291d198" volumeName="kubernetes.io/projected/82d520a7-fab4-46a7-bed2-9d4b3291d198-kube-api-access-d59s2" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455413 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455426 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6c9d77e-0f26-4a74-a08b-a2aff00389d2" volumeName="kubernetes.io/projected/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-kube-api-access-x2d7k" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455437 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455448 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d851015-b321-4b76-b09f-45c307ecfb37" volumeName="kubernetes.io/secret/2d851015-b321-4b76-b09f-45c307ecfb37-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455458 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82d520a7-fab4-46a7-bed2-9d4b3291d198" volumeName="kubernetes.io/empty-dir/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-tuned" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455470 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a302be0c-ed10-4c2d-8673-debd526d4994" volumeName="kubernetes.io/secret/a302be0c-ed10-4c2d-8673-debd526d4994-cert" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455481 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b056a7-e321-4ce7-ab1e-0d2e2824869f" volumeName="kubernetes.io/projected/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-kube-api-access-pw4wt" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455492 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1441c08-c76a-4954-80c9-8c1a0a04ce9f" volumeName="kubernetes.io/configmap/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-metrics-client-ca" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455503 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25720c7c-d528-4902-a94a-f437589ae9c9" volumeName="kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-config-volume" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455514 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d131de0-3a07-4d70-99e7-7aedb4648f31" volumeName="kubernetes.io/projected/6d131de0-3a07-4d70-99e7-7aedb4648f31-kube-api-access-pqsxp" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455526 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890a45ca-fe6f-4c91-9fda-034924575914" volumeName="kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455538 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c680bdb-3c36-4584-b1ec-78ab49d07aa9" volumeName="kubernetes.io/empty-dir/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-tmp-dir" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455551 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6310b145-4c78-4f58-91d0-be7558e3b1cc" volumeName="kubernetes.io/projected/6310b145-4c78-4f58-91d0-be7558e3b1cc-kube-api-access-xr4k7" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455562 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6310b145-4c78-4f58-91d0-be7558e3b1cc" volumeName="kubernetes.io/secret/6310b145-4c78-4f58-91d0-be7558e3b1cc-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455574 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82d520a7-fab4-46a7-bed2-9d4b3291d198" volumeName="kubernetes.io/empty-dir/82d520a7-fab4-46a7-bed2-9d4b3291d198-tmp" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455588 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f" volumeName="kubernetes.io/configmap/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-serviceca" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455599 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b056a7-e321-4ce7-ab1e-0d2e2824869f" volumeName="kubernetes.io/configmap/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-iptables-alerter-script" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455611 6107 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c88061-9a64-4653-96d8-e40f15c7f60c" volumeName="kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455623 6107 reconstruct.go:97] "Volume reconstruction finished" Oct 31 19:17:08.456025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.455630 6107 reconciler.go:29] "Reconciler: start to sync state" Oct 31 19:17:08.477449 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.477406 6107 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 19:17:08.478818 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.478791 6107 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 19:17:08.479064 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.479051 6107 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 19:17:08.479138 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.479079 6107 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 19:17:08.479138 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:08.479113 6107 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 19:17:08.498893 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.498837 6107 generic.go:356] "Generic (PLEG): container finished" podID="890a45ca-fe6f-4c91-9fda-034924575914" containerID="db55014858bfc8cb3ca1a53377caa819477e60af9d8f8026e75be22f84612329" exitCode=0 Oct 31 19:17:08.498893 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.498873 6107 generic.go:356] "Generic (PLEG): container finished" podID="890a45ca-fe6f-4c91-9fda-034924575914" containerID="928096bdea19e27a21c4aaf240f0880c6dc96ee369f63768dd56ddd82afbe424" exitCode=0 Oct 31 19:17:08.498893 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.498899 6107 generic.go:356] "Generic (PLEG): container finished" podID="890a45ca-fe6f-4c91-9fda-034924575914" containerID="04c70e41a29b4f52626db39a39925788cf1d8367e22fe8853d3b29637192ce8c" exitCode=0 Oct 31 19:17:08.499089 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.498905 6107 generic.go:356] "Generic (PLEG): container finished" podID="890a45ca-fe6f-4c91-9fda-034924575914" containerID="2e59369e0298ba9e055dd3b38271729415abe7b7c8e76e252b082bc7e2487930" exitCode=0 Oct 31 19:17:08.499089 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.498912 6107 generic.go:356] "Generic (PLEG): container finished" podID="890a45ca-fe6f-4c91-9fda-034924575914" containerID="d63e43472eaa4547660bcafa605d4e3e38c2678bb7c657ed16763cc9e6ffd12f" exitCode=0 Oct 31 19:17:08.499089 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.498918 6107 generic.go:356] "Generic (PLEG): container finished" podID="890a45ca-fe6f-4c91-9fda-034924575914" containerID="72af8a74067443f6b078f0050416afaeb2cc3a8f50522c78ae557a694e865c97" exitCode=0 Oct 31 19:17:08.510896 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.510841 6107 generic.go:356] "Generic (PLEG): container finished" podID="25720c7c-d528-4902-a94a-f437589ae9c9" containerID="143614d224bcde8e6b9dd3dcecb9d5a2987446d1c10395305c426a181efc5f25" exitCode=0 Oct 31 19:17:08.521391 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.521365 6107 generic.go:356] "Generic (PLEG): container finished" podID="7cd4cda822aa913fd3b1dccd57c5cf54" containerID="56425514f58539f94d83e1c7b8457cd7d82d045dfdc773b5845ab0db68b76aae" exitCode=0 Oct 31 19:17:08.528853 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.528830 6107 generic.go:356] "Generic (PLEG): container finished" podID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerID="12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58" exitCode=0 Oct 31 19:17:08.531162 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.531138 6107 generic.go:356] "Generic (PLEG): container finished" podID="6d131de0-3a07-4d70-99e7-7aedb4648f31" containerID="d2acaf63d0793f9762c91720ff2afe25d2344656a98637fb3a366bfe3756c3fa" exitCode=0 Oct 31 19:17:08.548089 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:08.548056 6107 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:17:08.554908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.554815 6107 manager.go:324] Recovery completed Oct 31 19:17:08.573695 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.573677 6107 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:17:08.575840 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.575820 6107 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:17:08.575953 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.575858 6107 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:17:08.575953 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.575874 6107 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:17:08.579175 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579150 6107 cpu_manager.go:226] "Starting" policy="none" Oct 31 19:17:08.579175 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579172 6107 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 19:17:08.579317 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579193 6107 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 19:17:08.579393 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579376 6107 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 19:17:08.579436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579396 6107 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 19:17:08.579436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579422 6107 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 19:17:08.579436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579429 6107 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 19:17:08.579436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579437 6107 policy_none.go:50] "Start" Oct 31 19:17:08.579589 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579461 6107 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 19:17:08.579589 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579474 6107 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 19:17:08.579702 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579687 6107 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 19:17:08.579745 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579705 6107 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 19:17:08.579745 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.579714 6107 policy_none.go:44] "Start" Oct 31 19:17:08.580201 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.580184 6107 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:17:08.580286 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:08.580150 6107 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 19:17:08.581278 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.581263 6107 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:17:08.581355 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.581289 6107 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:17:08.581355 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.581299 6107 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:17:08.584999 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.584975 6107 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 19:17:08.588312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.588297 6107 manager.go:341] "Starting Device Plugin manager" Oct 31 19:17:08.588388 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:08.588330 6107 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 19:17:08.588388 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.588342 6107 server.go:87] "Starting device plugin registration server" Oct 31 19:17:08.588650 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.588632 6107 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 19:17:08.588715 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.588652 6107 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 19:17:08.588852 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.588836 6107 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 19:17:08.588969 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.588937 6107 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 19:17:08.589025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.588976 6107 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 19:17:08.589025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.588983 6107 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 19:17:08.589139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.589059 6107 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T19:17:08.588966484Z","UUID":"9de79d24-164b-4908-90d2-e8d584eab7ef","Handler":null,"Name":"","Endpoint":""} Oct 31 19:17:08.590625 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:08.590605 6107 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 19:17:08.590708 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:08.590643 6107 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-164.us-west-2.compute.internal\" not found" Oct 31 19:17:08.591127 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.591113 6107 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 19:17:08.591191 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.591132 6107 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 19:17:08.631960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.631936 6107 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:17:08.632107 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.632088 6107 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:17:08.632176 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.632158 6107 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 19:17:08.632478 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.632462 6107 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:17:08.688977 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.688949 6107 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:17:08.690273 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.690254 6107 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:17:08.690376 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.690289 6107 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:17:08.690376 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.690304 6107 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:17:08.690376 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.690350 6107 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:08.721635 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.721591 6107 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:17:08.780553 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.780496 6107 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal"] Oct 31 19:17:08.781222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.781184 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" event={"ID":"7cd4cda822aa913fd3b1dccd57c5cf54","Type":"ContainerStarted","Data":"d0c8e31d12262fa034e83abd08c3b9241481b5efed0a26df44d7660aeaa0cce1"} Oct 31 19:17:08.781340 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.781227 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" event={"ID":"7cd4cda822aa913fd3b1dccd57c5cf54","Type":"ContainerDied","Data":"56425514f58539f94d83e1c7b8457cd7d82d045dfdc773b5845ab0db68b76aae"} Oct 31 19:17:08.781340 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.781244 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" event={"ID":"7cd4cda822aa913fd3b1dccd57c5cf54","Type":"ContainerStarted","Data":"b529eed16eeea08a460c5cbc2490192432094395ebf6824840bc6cf8fbd64c02"} Oct 31 19:17:08.781340 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.781248 6107 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:08.781340 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.781262 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" event={"ID":"3d2e8e59ab02a8ff64cf5b536f467713","Type":"ContainerStarted","Data":"ae9cf82de1dbef56205b7393cc6aa2c7ae83f4cbc100a232d24b17a91d88d315"} Oct 31 19:17:08.781340 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.781278 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" event={"ID":"3d2e8e59ab02a8ff64cf5b536f467713","Type":"ContainerStarted","Data":"d64471645d3a8476fc189257c351d44f1801c4de7558ebe4f9b8c740bbd3617e"} Oct 31 19:17:08.781340 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.781351 6107 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:08.790527 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.790505 6107 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:08.790612 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.790563 6107 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:08.841045 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.840969 6107 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:17:08.841045 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:08.841024 6107 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:08.841594 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.841575 6107 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:17:08.841690 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:08.841637 6107 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:08.933472 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.933442 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7cd4cda822aa913fd3b1dccd57c5cf54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"7cd4cda822aa913fd3b1dccd57c5cf54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:08.933472 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.933472 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cd4cda822aa913fd3b1dccd57c5cf54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"7cd4cda822aa913fd3b1dccd57c5cf54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:08.933699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:08.933489 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d2e8e59ab02a8ff64cf5b536f467713-config\") pod \"kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"3d2e8e59ab02a8ff64cf5b536f467713\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:09.034079 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.034046 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7cd4cda822aa913fd3b1dccd57c5cf54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"7cd4cda822aa913fd3b1dccd57c5cf54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:09.034244 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.034076 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7cd4cda822aa913fd3b1dccd57c5cf54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"7cd4cda822aa913fd3b1dccd57c5cf54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:09.034244 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.034112 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cd4cda822aa913fd3b1dccd57c5cf54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"7cd4cda822aa913fd3b1dccd57c5cf54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:09.034244 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.034139 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d2e8e59ab02a8ff64cf5b536f467713-config\") pod \"kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"3d2e8e59ab02a8ff64cf5b536f467713\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:09.034244 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.034185 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d2e8e59ab02a8ff64cf5b536f467713-config\") pod \"kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"3d2e8e59ab02a8ff64cf5b536f467713\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:09.034244 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.034201 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cd4cda822aa913fd3b1dccd57c5cf54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" (UID: \"7cd4cda822aa913fd3b1dccd57c5cf54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:09.434323 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.434288 6107 apiserver.go:52] "Watching apiserver" Oct 31 19:17:09.542612 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.542586 6107 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:09.542773 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.542703 6107 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:09.575344 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.575319 6107 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 19:17:09.575947 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.575924 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-k8lkj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl","openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f","openshift-image-registry/node-ca-dzg4p","openshift-monitoring/alertmanager-main-0","openshift-insights/insights-runtime-extractor-zxkn8","openshift-monitoring/node-exporter-2hfw5","kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal","openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s","openshift-cluster-node-tuning-operator/tuned-xk848","openshift-dns/node-resolver-h89v4","openshift-multus/multus-additional-cni-plugins-xxslf","openshift-ovn-kubernetes/ovnkube-node-s42ch","openshift-dns/dns-default-z4stw","kube-system/konnectivity-agent-szn4c","openshift-ingress-canary/ingress-canary-4kmqj","openshift-multus/multus-tdfnn","openshift-multus/network-metrics-daemon-f72jl","openshift-network-diagnostics/network-check-target-7lnww","openshift-monitoring/prometheus-k8s-0","kube-system/global-pull-secret-syncer-9hfm2"] Oct 31 19:17:09.598078 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.598044 6107 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:17:09.598207 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:09.598110 6107 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:09.598596 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.598582 6107 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:17:09.598665 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:09.598621 6107 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-164.us-west-2.compute.internal" Oct 31 19:17:09.626526 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.626500 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qh5p\"" Oct 31 19:17:09.626526 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.626511 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lwj7k\"" Oct 31 19:17:09.626675 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.626660 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 19:17:09.626729 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.626697 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgqq8\"" Oct 31 19:17:09.626729 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.626714 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tgkwh\"" Oct 31 19:17:09.626835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.626726 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 19:17:09.626835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.626752 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 19:17:09.626835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.626766 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 19:17:09.626835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.626751 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 19:17:09.627047 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.626956 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 19:17:09.627047 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.627008 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.627413 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.627460 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z8492\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.627520 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.627532 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.627571 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d7wnz\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.627693 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.627541 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.628044 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.628092 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.628247 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.628268 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.628364 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.628480 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vgfpk\"" Oct 31 19:17:09.628530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.628517 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zsjk9\"" Oct 31 19:17:09.629248 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.628526 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 19:17:09.629248 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.628592 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 19:17:09.629248 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.628814 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:17:09.629248 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.628853 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 19:17:09.629248 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.629009 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 19:17:09.629493 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.629477 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 19:17:09.629587 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.629569 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 19:17:09.629719 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.629707 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hlrbg\"" Oct 31 19:17:09.650208 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.650186 6107 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 19:17:09.726613 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.726581 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 19:17:09.726785 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.726679 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 19:17:09.726785 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.726686 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 19:17:09.726785 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.726698 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 19:17:09.726785 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.726680 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 19:17:09.726785 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.726708 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 19:17:09.726785 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.726687 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 19:17:09.727092 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.726922 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 19:17:09.727142 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727094 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 19:17:09.727142 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727131 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 19:17:09.727228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727142 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 19:17:09.727228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727194 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 19:17:09.727228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727205 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c47hb7itrpuc4\"" Oct 31 19:17:09.727228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727212 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mhxbx\"" Oct 31 19:17:09.727228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727222 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 19:17:09.727433 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727237 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 19:17:09.727433 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727273 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4ftk8\"" Oct 31 19:17:09.727433 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727274 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 19:17:09.727433 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727305 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 19:17:09.727433 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727322 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgzhm\"" Oct 31 19:17:09.727433 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727348 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mcs56\"" Oct 31 19:17:09.727433 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727366 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 19:17:09.727433 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727369 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4b89l\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727463 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727465 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727464 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727491 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rqvd7\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727496 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727496 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mvdsq\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727550 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727571 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727577 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tqdrb\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727609 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2kswh\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727618 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727612 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qq6mx\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727644 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 19:17:09.727743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727736 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727764 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727798 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727825 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727832 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727932 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727967 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.727971 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-m972p\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728064 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728105 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728139 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728166 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728181 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728279 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 19:17:09.728300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728289 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 19:17:09.728661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728345 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 19:17:09.728661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728353 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 19:17:09.728661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728376 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 19:17:09.728661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728373 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 19:17:09.728661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728372 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 19:17:09.728661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728550 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 19:17:09.728661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728596 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 19:17:09.728928 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728915 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 19:17:09.728928 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.728920 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 19:17:09.738153 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738131 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6310b145-4c78-4f58-91d0-be7558e3b1cc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:17:09.738246 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738160 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-host-slash\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:17:09.738246 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738175 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-var-lib-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.738246 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738207 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r6m7\" (UniqueName: \"kubernetes.io/projected/2d851015-b321-4b76-b09f-45c307ecfb37-kube-api-access-7r6m7\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:17:09.738361 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738317 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.738407 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738361 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-cni-multus\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.738407 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738392 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-etc-kubernetes\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.738504 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738418 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-systemd\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.738504 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738438 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6310b145-4c78-4f58-91d0-be7558e3b1cc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:17:09.738504 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738441 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-sys\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.738504 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738497 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-config-volume\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.738658 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738535 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-cni-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.738658 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738572 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.738658 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738631 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-k8s-cni-cncf-io\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.738835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738671 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-multus-certs\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.738835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738691 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-config-volume\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.738835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738707 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-accelerators-collector-config\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.738835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738741 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-device-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.738835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738836 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-kubernetes\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.739068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738875 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-env-overrides\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.739068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738913 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d851015-b321-4b76-b09f-45c307ecfb37-crio-socket\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:17:09.739068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738936 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.739068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.738957 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-accelerators-collector-config\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.739068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739005 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.739222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739067 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62dqg\" (UniqueName: \"kubernetes.io/projected/890a45ca-fe6f-4c91-9fda-034924575914-kube-api-access-62dqg\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.739222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739078 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-env-overrides\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.739222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739125 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz44\" (UniqueName: \"kubernetes.io/projected/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-kube-api-access-6jz44\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:17:09.739222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739146 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysctl-conf\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.739222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739202 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.739426 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739241 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d851015-b321-4b76-b09f-45c307ecfb37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:17:09.739426 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739320 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3de0cf07-8e8a-47c6-a382-bc56884867e6-cni-binary-copy\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.739508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739425 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-kubelet\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.739508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739449 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d851015-b321-4b76-b09f-45c307ecfb37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:17:09.739508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739460 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.739508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739488 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0875c4d7-7bfd-45ab-ae34-75e20e327605-tmp-dir\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:17:09.739508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739491 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3de0cf07-8e8a-47c6-a382-bc56884867e6-cni-binary-copy\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.739693 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739512 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-host\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:17:09.739693 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739530 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:17:09.739693 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739629 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.739693 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739660 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysconfig\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.739693 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739672 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0875c4d7-7bfd-45ab-ae34-75e20e327605-tmp-dir\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:17:09.739693 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739686 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-node-log\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.739896 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739698 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-metrics-certs\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:17:09.739896 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739704 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2d7k\" (UniqueName: \"kubernetes.io/projected/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-kube-api-access-x2d7k\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.739896 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739810 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-tmp-dir\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:17:09.739896 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739863 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.740051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739901 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-config\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.740051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739926 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.740051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739940 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-tmp-dir\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:17:09.740051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739951 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-cnibin\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.740051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739976 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-web-config\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.740051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.739998 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a302be0c-ed10-4c2d-8673-debd526d4994-cert\") pod \"ingress-canary-4kmqj\" (UID: \"a302be0c-ed10-4c2d-8673-debd526d4994\") " pod="openshift-ingress-canary/ingress-canary-4kmqj" Oct 31 19:17:09.740051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740022 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/474d47d1-f185-4127-a085-786c6ada462e-konnectivity-ca\") pod \"konnectivity-agent-szn4c\" (UID: \"474d47d1-f185-4127-a085-786c6ada462e\") " pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:17:09.740051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740045 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z96kc\" (UniqueName: \"kubernetes.io/projected/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-kube-api-access-z96kc\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:17:09.740051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740046 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.740367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740068 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlppw\" (UniqueName: \"kubernetes.io/projected/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-kube-api-access-zlppw\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:17:09.740367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740092 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-config\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.740367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740161 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.740367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740198 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-web-config\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.740367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740234 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d851015-b321-4b76-b09f-45c307ecfb37-data-volume\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:17:09.740367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740264 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a302be0c-ed10-4c2d-8673-debd526d4994-cert\") pod \"ingress-canary-4kmqj\" (UID: \"a302be0c-ed10-4c2d-8673-debd526d4994\") " pod="openshift-ingress-canary/ingress-canary-4kmqj" Oct 31 19:17:09.740367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740257 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:17:09.740367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740310 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.740367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740328 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d851015-b321-4b76-b09f-45c307ecfb37-data-volume\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:17:09.740367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740337 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.740367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740366 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740384 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwl2\" (UniqueName: \"kubernetes.io/projected/a302be0c-ed10-4c2d-8673-debd526d4994-kube-api-access-gtwl2\") pod \"ingress-canary-4kmqj\" (UID: \"a302be0c-ed10-4c2d-8673-debd526d4994\") " pod="openshift-ingress-canary/ingress-canary-4kmqj" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740390 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/474d47d1-f185-4127-a085-786c6ada462e-konnectivity-ca\") pod \"konnectivity-agent-szn4c\" (UID: \"474d47d1-f185-4127-a085-786c6ada462e\") " pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740425 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740429 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6310b145-4c78-4f58-91d0-be7558e3b1cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740452 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f947add9-2413-4c26-860a-a5431e7cd2b9-kubelet-config\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740471 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-slash\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740509 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740538 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740545 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740575 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740580 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740629 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6310b145-4c78-4f58-91d0-be7558e3b1cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740658 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-cnibin\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740675 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740690 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngplm\" (UniqueName: \"kubernetes.io/projected/0875c4d7-7bfd-45ab-ae34-75e20e327605-kube-api-access-ngplm\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740727 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4wt\" (UniqueName: \"kubernetes.io/projected/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-kube-api-access-pw4wt\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740765 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.740835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740785 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-systemd-units\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740908 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rps8q\" (UniqueName: \"kubernetes.io/projected/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-api-access-rps8q\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.740974 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741041 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741058 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82d520a7-fab4-46a7-bed2-9d4b3291d198-tmp\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741074 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-kubelet\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741091 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr4k7\" (UniqueName: \"kubernetes.io/projected/6310b145-4c78-4f58-91d0-be7558e3b1cc-kube-api-access-xr4k7\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741109 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-run\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741091 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741151 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-host\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741213 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82d520a7-fab4-46a7-bed2-9d4b3291d198-tmp\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741222 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-netns\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741241 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741274 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741303 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741332 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741338 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741363 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741401 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysctl-d\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741418 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq5n\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-kube-api-access-mtq5n\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741426 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741506 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-root\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741533 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f947add9-2413-4c26-860a-a5431e7cd2b9-dbus\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741542 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741574 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-etc-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741615 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovnkube-config\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.741642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741651 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f947add9-2413-4c26-860a-a5431e7cd2b9-original-pull-secret\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741653 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-system-cni-dir\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741687 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-sys-fs\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741703 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-lib-modules\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741751 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741782 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741793 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovnkube-config\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741804 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-systemd\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741830 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741891 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-config-volume\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741934 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-var-lib-kubelet\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741953 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741951 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.741994 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovn-node-metrics-cert\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742002 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742048 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742178 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-config-volume\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742193 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-textfile\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742206 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovn-node-metrics-cert\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742215 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqsxp\" (UniqueName: \"kubernetes.io/projected/6d131de0-3a07-4d70-99e7-7aedb4648f31-kube-api-access-pqsxp\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742223 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742232 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-ovn\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742257 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742263 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-textfile\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742281 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742317 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-hostroot\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742348 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742360 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-etc-selinux\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742407 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d59s2\" (UniqueName: \"kubernetes.io/projected/82d520a7-fab4-46a7-bed2-9d4b3291d198-kube-api-access-d59s2\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742436 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-log-socket\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742468 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742516 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742542 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-config-out\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742548 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742591 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-web-config\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.742648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742620 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-daemon-config\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742709 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-config-out\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742716 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-iptables-alerter-script\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742737 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742742 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-metrics-tls\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742770 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-sys\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742783 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-web-config\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742787 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742829 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742905 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9k9l\" (UniqueName: \"kubernetes.io/projected/3de0cf07-8e8a-47c6-a382-bc56884867e6-kube-api-access-f9k9l\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742920 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-iptables-alerter-script\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742939 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742941 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-metrics-tls\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742970 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6310b145-4c78-4f58-91d0-be7558e3b1cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742971 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.742998 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-cni-netd\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743048 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-daemon-config\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743053 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743090 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743092 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4gh\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-kube-api-access-vj4gh\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743133 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743133 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6310b145-4c78-4f58-91d0-be7558e3b1cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743160 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743180 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743189 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-tls\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743226 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-modprobe-d\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743232 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743245 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-config-out\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743273 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743279 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-wtmp\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743317 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743340 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwm6\" (UniqueName: \"kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6\") pod \"network-check-target-7lnww\" (UID: \"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe\") " pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743356 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d851015-b321-4b76-b09f-45c307ecfb37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743352 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-config-out\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743378 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-tls\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743392 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-system-cni-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743421 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-cni-bin\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743445 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-conf-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743469 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0875c4d7-7bfd-45ab-ae34-75e20e327605-hosts-file\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743501 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d851015-b321-4b76-b09f-45c307ecfb37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743517 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-serviceca\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743545 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-run-netns\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743563 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovnkube-script-lib\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743577 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743612 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpck4\" (UniqueName: \"kubernetes.io/projected/8d3cf005-2850-4952-b7ed-7aeeb299603e-kube-api-access-bpck4\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743628 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-tuned\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743645 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-cni-bin\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743658 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-socket-dir-parent\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743674 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743726 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-registration-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743745 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743756 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/474d47d1-f185-4127-a085-786c6ada462e-agent-certs\") pod \"konnectivity-agent-szn4c\" (UID: \"474d47d1-f185-4127-a085-786c6ada462e\") " pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743783 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-ovnkube-script-lib\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743794 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743814 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-tuned\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743820 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743851 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d131de0-3a07-4d70-99e7-7aedb4648f31-metrics-client-ca\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743860 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/890a45ca-fe6f-4c91-9fda-034924575914-cni-binary-copy\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743898 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-socket-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.743928 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:17:09.744014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.744039 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.745577 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.744067 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d131de0-3a07-4d70-99e7-7aedb4648f31-metrics-client-ca\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.745577 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.744045 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:17:09.745577 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.744106 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.745577 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.744148 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/474d47d1-f185-4127-a085-786c6ada462e-agent-certs\") pod \"konnectivity-agent-szn4c\" (UID: \"474d47d1-f185-4127-a085-786c6ada462e\") " pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:17:09.745577 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.744339 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-serviceca\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:17:09.774468 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.774432 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 19:17:09.774903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.774862 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 19:17:09.774903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.774894 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 19:17:09.780853 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.780834 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.780928 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.780913 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.783831 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.783810 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.789423 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.789406 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r6m7\" (UniqueName: \"kubernetes.io/projected/2d851015-b321-4b76-b09f-45c307ecfb37-kube-api-access-7r6m7\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:17:09.791398 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.791376 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2d7k\" (UniqueName: \"kubernetes.io/projected/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-kube-api-access-x2d7k\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.791470 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.791424 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z96kc\" (UniqueName: \"kubernetes.io/projected/9c680bdb-3c36-4584-b1ec-78ab49d07aa9-kube-api-access-z96kc\") pod \"dns-default-z4stw\" (UID: \"9c680bdb-3c36-4584-b1ec-78ab49d07aa9\") " pod="openshift-dns/dns-default-z4stw" Oct 31 19:17:09.791625 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.791607 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlppw\" (UniqueName: \"kubernetes.io/projected/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-kube-api-access-zlppw\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:17:09.791730 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.791716 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jz44\" (UniqueName: \"kubernetes.io/projected/d27497f4-b29e-46c2-8f63-664fbe7e8dd1-kube-api-access-6jz44\") pod \"network-metrics-daemon-f72jl\" (UID: \"d27497f4-b29e-46c2-8f63-664fbe7e8dd1\") " pod="openshift-multus/network-metrics-daemon-f72jl" Oct 31 19:17:09.791763 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.791752 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62dqg\" (UniqueName: \"kubernetes.io/projected/890a45ca-fe6f-4c91-9fda-034924575914-kube-api-access-62dqg\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.793848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.793830 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rps8q\" (UniqueName: \"kubernetes.io/projected/e1441c08-c76a-4954-80c9-8c1a0a04ce9f-kube-api-access-rps8q\") pod \"kube-state-metrics-5dc8d7d89b-hd95f\" (UID: \"e1441c08-c76a-4954-80c9-8c1a0a04ce9f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hd95f" Oct 31 19:17:09.793946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.793835 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwl2\" (UniqueName: \"kubernetes.io/projected/a302be0c-ed10-4c2d-8673-debd526d4994-kube-api-access-gtwl2\") pod \"ingress-canary-4kmqj\" (UID: \"a302be0c-ed10-4c2d-8673-debd526d4994\") " pod="openshift-ingress-canary/ingress-canary-4kmqj" Oct 31 19:17:09.794043 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.794027 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngplm\" (UniqueName: \"kubernetes.io/projected/0875c4d7-7bfd-45ab-ae34-75e20e327605-kube-api-access-ngplm\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:17:09.794163 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.794144 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4wt\" (UniqueName: \"kubernetes.io/projected/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-kube-api-access-pw4wt\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:17:09.794540 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.794525 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr4k7\" (UniqueName: \"kubernetes.io/projected/6310b145-4c78-4f58-91d0-be7558e3b1cc-kube-api-access-xr4k7\") pod \"openshift-state-metrics-7bd7489957-n7m2s\" (UID: \"6310b145-4c78-4f58-91d0-be7558e3b1cc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-n7m2s" Oct 31 19:17:09.795435 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.795417 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqsxp\" (UniqueName: \"kubernetes.io/projected/6d131de0-3a07-4d70-99e7-7aedb4648f31-kube-api-access-pqsxp\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.795534 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.795518 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d59s2\" (UniqueName: \"kubernetes.io/projected/82d520a7-fab4-46a7-bed2-9d4b3291d198-kube-api-access-d59s2\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.795819 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.795801 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9k9l\" (UniqueName: \"kubernetes.io/projected/3de0cf07-8e8a-47c6-a382-bc56884867e6-kube-api-access-f9k9l\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.795925 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.795909 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtq5n\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-kube-api-access-mtq5n\") pod \"alertmanager-main-0\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:09.795969 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.795938 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpck4\" (UniqueName: \"kubernetes.io/projected/8d3cf005-2850-4952-b7ed-7aeeb299603e-kube-api-access-bpck4\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.796626 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.796608 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcwm6\" (UniqueName: \"kubernetes.io/projected/64d8b12c-0fc1-4b11-9a11-8155f18c8dbe-kube-api-access-vcwm6\") pod \"network-check-target-7lnww\" (UID: \"64d8b12c-0fc1-4b11-9a11-8155f18c8dbe\") " pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:17:09.797127 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.797113 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4gh\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-kube-api-access-vj4gh\") pod \"prometheus-k8s-0\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:09.848529 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848495 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-netns\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.848529 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848527 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysctl-d\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848543 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-root\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848560 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f947add9-2413-4c26-860a-a5431e7cd2b9-dbus\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848581 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-etc-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848605 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-system-cni-dir\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848611 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-netns\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848619 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-root\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848629 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-sys-fs\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848650 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-lib-modules\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848664 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f947add9-2413-4c26-860a-a5431e7cd2b9-dbus\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848683 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-etc-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848675 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-systemd\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848698 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-system-cni-dir\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848718 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-var-lib-kubelet\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848721 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-systemd\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848736 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysctl-d\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848746 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-sys-fs\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848743 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848779 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848785 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-lib-modules\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848792 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-ovn\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848793 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-var-lib-kubelet\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848817 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-hostroot\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848832 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-etc-selinux\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.848848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848850 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-log-socket\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848873 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848907 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-sys\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848906 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-run-ovn\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848917 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-etc-selinux\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848942 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-cni-netd\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848956 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-hostroot\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848960 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-log-socket\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848964 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848987 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-modprobe-d\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848991 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-sys\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.848995 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849011 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-wtmp\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849028 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-cni-netd\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849035 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849041 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849058 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-system-cni-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849080 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-modprobe-d\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849083 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d131de0-3a07-4d70-99e7-7aedb4648f31-node-exporter-wtmp\") pod \"node-exporter-2hfw5\" (UID: \"6d131de0-3a07-4d70-99e7-7aedb4648f31\") " pod="openshift-monitoring/node-exporter-2hfw5" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849082 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-cni-bin\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849092 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849114 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-cni-bin\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849118 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-conf-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849136 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0875c4d7-7bfd-45ab-ae34-75e20e327605-hosts-file\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849144 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-system-cni-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849151 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-run-netns\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849168 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-run-netns\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849176 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-cni-bin\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849194 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-conf-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849194 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-socket-dir-parent\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849213 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0875c4d7-7bfd-45ab-ae34-75e20e327605-hosts-file\") pod \"node-resolver-h89v4\" (UID: \"0875c4d7-7bfd-45ab-ae34-75e20e327605\") " pod="openshift-dns/node-resolver-h89v4" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849216 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-registration-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849223 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-socket-dir-parent\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849245 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-socket-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849253 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-cni-bin\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849261 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-host-slash\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849273 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-socket-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849282 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b056a7-e321-4ce7-ab1e-0d2e2824869f-host-slash\") pod \"iptables-alerter-k8lkj\" (UID: \"c8b056a7-e321-4ce7-ab1e-0d2e2824869f\") " pod="openshift-network-operator/iptables-alerter-k8lkj" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849276 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-var-lib-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849297 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-var-lib-openvswitch\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849245 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-registration-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849305 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-cni-multus\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849320 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-etc-kubernetes\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849340 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-etc-kubernetes\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849342 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-cni-multus\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849366 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-systemd\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849381 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-sys\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849394 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-cni-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849410 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-k8s-cni-cncf-io\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849433 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-multus-certs\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849444 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-sys\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849449 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-k8s-cni-cncf-io\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849459 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-device-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849465 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-multus-cni-dir\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849477 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-systemd\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849481 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-kubernetes\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849487 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-run-multus-certs\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849495 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d3cf005-2850-4952-b7ed-7aeeb299603e-device-dir\") pod \"aws-ebs-csi-driver-node-v7pgl\" (UID: \"8d3cf005-2850-4952-b7ed-7aeeb299603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v7pgl" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849501 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d851015-b321-4b76-b09f-45c307ecfb37-crio-socket\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849516 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849533 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysctl-conf\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849546 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-kubernetes\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849572 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d851015-b321-4b76-b09f-45c307ecfb37-crio-socket\") pod \"insights-runtime-extractor-zxkn8\" (UID: \"2d851015-b321-4b76-b09f-45c307ecfb37\") " pod="openshift-insights/insights-runtime-extractor-zxkn8" Oct 31 19:17:09.849546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849578 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849600 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-kubelet\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849640 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-host\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849661 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysconfig\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849666 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-host-var-lib-kubelet\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849677 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysctl-conf\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849682 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-node-log\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849685 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f-host\") pod \"node-ca-dzg4p\" (UID: \"9be4a9b0-a460-4e50-aa8a-961f5a3e4d7f\") " pod="openshift-image-registry/node-ca-dzg4p" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849712 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-etc-sysconfig\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849718 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-cnibin\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849747 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f947add9-2413-4c26-860a-a5431e7cd2b9-kubelet-config\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849761 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-slash\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849714 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-node-log\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849778 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-cnibin\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849802 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-systemd-units\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849814 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/890a45ca-fe6f-4c91-9fda-034924575914-cnibin\") pod \"multus-additional-cni-plugins-xxslf\" (UID: \"890a45ca-fe6f-4c91-9fda-034924575914\") " pod="openshift-multus/multus-additional-cni-plugins-xxslf" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849820 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f947add9-2413-4c26-860a-a5431e7cd2b9-kubelet-config\") pod \"global-pull-secret-syncer-9hfm2\" (UID: \"f947add9-2413-4c26-860a-a5431e7cd2b9\") " pod="kube-system/global-pull-secret-syncer-9hfm2" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849823 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-slash\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849838 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3de0cf07-8e8a-47c6-a382-bc56884867e6-cnibin\") pod \"multus-tdfnn\" (UID: \"3de0cf07-8e8a-47c6-a382-bc56884867e6\") " pod="openshift-multus/multus-tdfnn" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849846 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-systemd-units\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849865 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-kubelet\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849901 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-run\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849920 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-host\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849922 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6c9d77e-0f26-4a74-a08b-a2aff00389d2-host-kubelet\") pod \"ovnkube-node-s42ch\" (UID: \"f6c9d77e-0f26-4a74-a08b-a2aff00389d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849982 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-run\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:09.851228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:09.849983 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82d520a7-fab4-46a7-bed2-9d4b3291d198-host\") pod \"tuned-xk848\" (UID: \"82d520a7-fab4-46a7-bed2-9d4b3291d198\") " pod="openshift-cluster-node-tuning-operator/tuned-xk848" Oct 31 19:17:11.805704 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:11.805672 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:11.820516 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:11.820493 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:11.937270 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:11.937244 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:12.752062 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:12.752020 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z4stw" Oct 31 19:17:13.331677 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:13.331646 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:13.332050 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:13.331941 6107 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 19:17:15.323873 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:15.323849 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s42ch" Oct 31 19:17:19.337697 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:19.337594 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:17:20.401720 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:20.401686 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:17:28.992188 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:28.992159 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-szn4c" Oct 31 19:17:39.340812 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:39.340781 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7lnww" Oct 31 19:17:40.574972 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:40.574936 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 19:17:40.575489 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:40.575438 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="alertmanager" containerID="cri-o://782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8" gracePeriod=120 Oct 31 19:17:40.575568 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:40.575495 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="kube-rbac-proxy" containerID="cri-o://02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72" gracePeriod=120 Oct 31 19:17:40.575568 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:40.575516 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="prom-label-proxy" containerID="cri-o://11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c" gracePeriod=120 Oct 31 19:17:40.575661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:40.575533 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="kube-rbac-proxy-web" containerID="cri-o://4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5" gracePeriod=120 Oct 31 19:17:40.575661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:40.575550 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="kube-rbac-proxy-metric" containerID="cri-o://6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85" gracePeriod=120 Oct 31 19:17:40.575917 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:40.575857 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="config-reloader" containerID="cri-o://28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1" gracePeriod=120 Oct 31 19:17:40.591762 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:40.591743 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:41.656342 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:41.656309 6107 generic.go:356] "Generic (PLEG): container finished" podID="25720c7c-d528-4902-a94a-f437589ae9c9" containerID="11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c" exitCode=0 Oct 31 19:17:41.656342 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:41.656337 6107 generic.go:356] "Generic (PLEG): container finished" podID="25720c7c-d528-4902-a94a-f437589ae9c9" containerID="02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72" exitCode=0 Oct 31 19:17:41.656342 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:41.656344 6107 generic.go:356] "Generic (PLEG): container finished" podID="25720c7c-d528-4902-a94a-f437589ae9c9" containerID="4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5" exitCode=0 Oct 31 19:17:41.656342 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:41.656350 6107 generic.go:356] "Generic (PLEG): container finished" podID="25720c7c-d528-4902-a94a-f437589ae9c9" containerID="28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1" exitCode=0 Oct 31 19:17:41.656799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:41.656356 6107 generic.go:356] "Generic (PLEG): container finished" podID="25720c7c-d528-4902-a94a-f437589ae9c9" containerID="782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8" exitCode=0 Oct 31 19:17:41.656799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:41.656376 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerDied","Data":"11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c"} Oct 31 19:17:41.656799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:41.656463 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerDied","Data":"02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72"} Oct 31 19:17:41.656799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:41.656491 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerDied","Data":"4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5"} Oct 31 19:17:41.656799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:41.656515 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerDied","Data":"28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1"} Oct 31 19:17:41.656799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:41.656536 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerDied","Data":"782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8"} Oct 31 19:17:42.354320 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.354294 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.388818 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.388793 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-tls-assets\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-tls-assets\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.388998 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.388828 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-trusted-ca-bundle\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.388998 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.388868 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-cluster-tls-config\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.388998 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.388918 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-config-out\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.388998 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.388952 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.388998 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.388992 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.389271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.389020 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-kube-api-access-mtq5n\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-kube-api-access-mtq5n\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.389271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.389054 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-web-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-web-config\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.389271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.389099 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-main-db\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.389271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.389155 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-metrics-client-ca\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.389271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.389210 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-web\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.389271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.389226 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-trusted-ca-bundle" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:42.389271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.389254 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-main-tls\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.389635 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.389283 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-config-volume\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-config-volume\") pod \"25720c7c-d528-4902-a94a-f437589ae9c9\" (UID: \"25720c7c-d528-4902-a94a-f437589ae9c9\") " Oct 31 19:17:42.389635 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.389499 6107 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.390053 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.390025 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-metrics-client-ca" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:42.390350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.390327 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-main-db" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:17:42.392706 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.392476 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-kube-api-access-mtq5n" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "kube-api-access-mtq5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:17:42.392805 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.392756 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-metric" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:42.393014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.392987 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-tls-assets" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:17:42.393500 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.393432 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:42.393500 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.393465 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-config-out" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:17:42.393673 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.393653 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-config-volume" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:42.393737 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.393722 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-web" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:42.394767 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.394744 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-main-tls" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:42.396389 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.396371 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-cluster-tls-config" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:42.402565 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.402540 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-web-config" pod "25720c7c-d528-4902-a94a-f437589ae9c9" (UID: "25720c7c-d528-4902-a94a-f437589ae9c9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:42.490021 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.489995 6107 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-cluster-tls-config\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.490193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.490025 6107 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-config-out\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.490193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.490040 6107 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.490193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.490054 6107 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.490193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.490068 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtq5n\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-kube-api-access-mtq5n\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.490193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.490080 6107 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-web-config\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.490193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.490094 6107 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25720c7c-d528-4902-a94a-f437589ae9c9-alertmanager-main-db\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.490193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.490107 6107 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25720c7c-d528-4902-a94a-f437589ae9c9-metrics-client-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.490193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.490119 6107 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.490193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.490132 6107 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-secret-alertmanager-main-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.490193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.490145 6107 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25720c7c-d528-4902-a94a-f437589ae9c9-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.490193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.490157 6107 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25720c7c-d528-4902-a94a-f437589ae9c9-tls-assets\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:42.663834 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.663797 6107 generic.go:356] "Generic (PLEG): container finished" podID="25720c7c-d528-4902-a94a-f437589ae9c9" containerID="6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85" exitCode=0 Oct 31 19:17:42.664205 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.663869 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerDied","Data":"6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85"} Oct 31 19:17:42.664205 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.663951 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25720c7c-d528-4902-a94a-f437589ae9c9","Type":"ContainerDied","Data":"40f3088e96e3723af2d48668d39a9bc14af3f66038c105289ef1a00eaf7d60b8"} Oct 31 19:17:42.664205 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.663985 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.664205 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.663986 6107 scope.go:122] "RemoveContainer" containerID="11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c" Oct 31 19:17:42.673448 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.673431 6107 scope.go:122] "RemoveContainer" containerID="6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85" Oct 31 19:17:42.680027 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.680008 6107 scope.go:122] "RemoveContainer" containerID="02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72" Oct 31 19:17:42.687561 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.687543 6107 scope.go:122] "RemoveContainer" containerID="4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5" Oct 31 19:17:42.694384 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.694361 6107 scope.go:122] "RemoveContainer" containerID="28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1" Oct 31 19:17:42.700842 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.700824 6107 scope.go:122] "RemoveContainer" containerID="782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8" Oct 31 19:17:42.707207 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.707190 6107 scope.go:122] "RemoveContainer" containerID="143614d224bcde8e6b9dd3dcecb9d5a2987446d1c10395305c426a181efc5f25" Oct 31 19:17:42.713309 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.713292 6107 scope.go:122] "RemoveContainer" containerID="11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c" Oct 31 19:17:42.713555 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:42.713534 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c\": container with ID starting with 11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c not found: ID does not exist" containerID="11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c" Oct 31 19:17:42.713620 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.713569 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c"} err="failed to get container status \"11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c\": rpc error: code = NotFound desc = could not find container \"11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c\": container with ID starting with 11e2751b96ae94eab41262cae7e23adb8017ecbc7022443d906ad7e27f520f2c not found: ID does not exist" Oct 31 19:17:42.713620 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.713607 6107 scope.go:122] "RemoveContainer" containerID="6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85" Oct 31 19:17:42.713911 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:42.713862 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85\": container with ID starting with 6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85 not found: ID does not exist" containerID="6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85" Oct 31 19:17:42.713959 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.713902 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85"} err="failed to get container status \"6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85\": rpc error: code = NotFound desc = could not find container \"6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85\": container with ID starting with 6611379c4718736148a29f025cb4e4a2f40c1c5f1e4bb26a566bb4c043ffae85 not found: ID does not exist" Oct 31 19:17:42.713959 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.713924 6107 scope.go:122] "RemoveContainer" containerID="02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72" Oct 31 19:17:42.714218 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:42.714194 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72\": container with ID starting with 02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72 not found: ID does not exist" containerID="02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72" Oct 31 19:17:42.714257 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.714226 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72"} err="failed to get container status \"02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72\": rpc error: code = NotFound desc = could not find container \"02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72\": container with ID starting with 02c907668779ffc1ccf218b6d0ab82c870f9d28dbb5a9ce29d40723aeac79c72 not found: ID does not exist" Oct 31 19:17:42.714257 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.714249 6107 scope.go:122] "RemoveContainer" containerID="4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5" Oct 31 19:17:42.714479 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:42.714464 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5\": container with ID starting with 4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5 not found: ID does not exist" containerID="4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5" Oct 31 19:17:42.714522 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.714484 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5"} err="failed to get container status \"4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5\": rpc error: code = NotFound desc = could not find container \"4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5\": container with ID starting with 4f4875c6f175774714669be379e7a59213c24cb305d97a177e13d0662391b7e5 not found: ID does not exist" Oct 31 19:17:42.714522 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.714498 6107 scope.go:122] "RemoveContainer" containerID="28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1" Oct 31 19:17:42.714710 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:42.714689 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1\": container with ID starting with 28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1 not found: ID does not exist" containerID="28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1" Oct 31 19:17:42.714751 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.714713 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1"} err="failed to get container status \"28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1\": rpc error: code = NotFound desc = could not find container \"28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1\": container with ID starting with 28606c4220bb606b95a136786897d8a4c38b3b93199fffa3e064a377358d7ea1 not found: ID does not exist" Oct 31 19:17:42.714751 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.714726 6107 scope.go:122] "RemoveContainer" containerID="782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8" Oct 31 19:17:42.714920 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:42.714903 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8\": container with ID starting with 782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8 not found: ID does not exist" containerID="782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8" Oct 31 19:17:42.714978 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.714925 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8"} err="failed to get container status \"782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8\": rpc error: code = NotFound desc = could not find container \"782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8\": container with ID starting with 782ba4b0178a39d255f6f8a38ffdc780e0ec429f9a5fe8528760ad861d2beda8 not found: ID does not exist" Oct 31 19:17:42.714978 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.714937 6107 scope.go:122] "RemoveContainer" containerID="143614d224bcde8e6b9dd3dcecb9d5a2987446d1c10395305c426a181efc5f25" Oct 31 19:17:42.715144 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:42.715128 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143614d224bcde8e6b9dd3dcecb9d5a2987446d1c10395305c426a181efc5f25\": container with ID starting with 143614d224bcde8e6b9dd3dcecb9d5a2987446d1c10395305c426a181efc5f25 not found: ID does not exist" containerID="143614d224bcde8e6b9dd3dcecb9d5a2987446d1c10395305c426a181efc5f25" Oct 31 19:17:42.715187 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.715147 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143614d224bcde8e6b9dd3dcecb9d5a2987446d1c10395305c426a181efc5f25"} err="failed to get container status \"143614d224bcde8e6b9dd3dcecb9d5a2987446d1c10395305c426a181efc5f25\": rpc error: code = NotFound desc = could not find container \"143614d224bcde8e6b9dd3dcecb9d5a2987446d1c10395305c426a181efc5f25\": container with ID starting with 143614d224bcde8e6b9dd3dcecb9d5a2987446d1c10395305c426a181efc5f25 not found: ID does not exist" Oct 31 19:17:42.826371 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.824527 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 19:17:42.829116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.829094 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 19:17:42.869527 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.869500 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 19:17:42.869994 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.869968 6107 cpu_manager.go:414] "containerMap: removing container" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="prom-label-proxy" Oct 31 19:17:42.869994 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.869993 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="prom-label-proxy" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870006 6107 cpu_manager.go:414] "containerMap: removing container" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="config-reloader" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870013 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="config-reloader" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870022 6107 cpu_manager.go:414] "containerMap: removing container" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="kube-rbac-proxy-metric" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870029 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="kube-rbac-proxy-metric" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870044 6107 cpu_manager.go:414] "containerMap: removing container" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="init-config-reloader" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870051 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="init-config-reloader" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870062 6107 cpu_manager.go:414] "containerMap: removing container" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="kube-rbac-proxy-web" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870068 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="kube-rbac-proxy-web" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870076 6107 cpu_manager.go:414] "containerMap: removing container" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="kube-rbac-proxy" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870083 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="kube-rbac-proxy" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870103 6107 cpu_manager.go:414] "containerMap: removing container" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="alertmanager" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870108 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="alertmanager" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870165 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="config-reloader" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870173 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="kube-rbac-proxy-metric" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870182 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="init-config-reloader" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870190 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="kube-rbac-proxy-web" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870197 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="kube-rbac-proxy" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870206 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="alertmanager" Oct 31 19:17:42.870228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.870217 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" containerName="prom-label-proxy" Oct 31 19:17:42.899669 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.899642 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.936588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.936560 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 19:17:42.948736 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.948710 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 19:17:42.948868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.948736 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 19:17:42.948868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.948737 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 19:17:42.948868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.948796 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 19:17:42.948868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.948825 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 19:17:42.948868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.948834 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 19:17:42.948868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.948857 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 19:17:42.949128 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.948863 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mhxbx\"" Oct 31 19:17:42.949128 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.949047 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 19:17:42.981805 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.981776 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 19:17:42.995022 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995001 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.995111 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995029 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a86c21-e504-455a-83d5-e68baf8191c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.995111 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995050 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.995111 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995067 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.995259 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995139 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.995259 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995242 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.995363 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995275 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45a86c21-e504-455a-83d5-e68baf8191c3-config-out\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.995363 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995304 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-web-config\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.995363 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995335 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.995464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995398 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/45a86c21-e504-455a-83d5-e68baf8191c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.995464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995454 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45a86c21-e504-455a-83d5-e68baf8191c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.995531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995491 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45a86c21-e504-455a-83d5-e68baf8191c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:42.995531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:42.995512 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxjjz\" (UniqueName: \"kubernetes.io/projected/45a86c21-e504-455a-83d5-e68baf8191c3-kube-api-access-qxjjz\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.095921 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.095834 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.095921 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.095865 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a86c21-e504-455a-83d5-e68baf8191c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.095921 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.095900 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.096169 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.096024 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.096169 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.096062 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.096169 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.096097 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.096169 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.096126 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45a86c21-e504-455a-83d5-e68baf8191c3-config-out\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.096169 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.096153 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-web-config\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.096392 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.096212 6107 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 19:17:43.096392 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.096282 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.096392 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.096323 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/45a86c21-e504-455a-83d5-e68baf8191c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.096724 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.096696 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45a86c21-e504-455a-83d5-e68baf8191c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.097008 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.096985 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a86c21-e504-455a-83d5-e68baf8191c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.097092 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.097041 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45a86c21-e504-455a-83d5-e68baf8191c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.097092 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.097072 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/45a86c21-e504-455a-83d5-e68baf8191c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.097092 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.097085 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxjjz\" (UniqueName: \"kubernetes.io/projected/45a86c21-e504-455a-83d5-e68baf8191c3-kube-api-access-qxjjz\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.097741 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.097716 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45a86c21-e504-455a-83d5-e68baf8191c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.099285 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.099264 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.099907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.099602 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45a86c21-e504-455a-83d5-e68baf8191c3-config-out\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.099907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.099723 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.099907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.099754 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.100107 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.099946 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-web-config\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.100107 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.099960 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.100107 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.100039 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.100107 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.100078 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/45a86c21-e504-455a-83d5-e68baf8191c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.100817 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.100800 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45a86c21-e504-455a-83d5-e68baf8191c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.150538 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.150511 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxjjz\" (UniqueName: \"kubernetes.io/projected/45a86c21-e504-455a-83d5-e68baf8191c3-kube-api-access-qxjjz\") pod \"alertmanager-main-0\" (UID: \"45a86c21-e504-455a-83d5-e68baf8191c3\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.303929 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.303874 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 19:17:43.742999 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:43.742963 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 19:17:43.746806 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:43.746768 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a86c21_e504_455a_83d5_e68baf8191c3.slice/crio-6234f59a956f776be1d08a605e105d4cda22e5e8f084d29aedbe51fa79392954 WatchSource:0}: Error finding container 6234f59a956f776be1d08a605e105d4cda22e5e8f084d29aedbe51fa79392954: Status 404 returned error can't find the container with id 6234f59a956f776be1d08a605e105d4cda22e5e8f084d29aedbe51fa79392954 Oct 31 19:17:44.483668 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:44.483624 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25720c7c-d528-4902-a94a-f437589ae9c9" path="/var/lib/kubelet/pods/25720c7c-d528-4902-a94a-f437589ae9c9/volumes" Oct 31 19:17:44.672016 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:44.671980 6107 generic.go:356] "Generic (PLEG): container finished" podID="45a86c21-e504-455a-83d5-e68baf8191c3" containerID="f4a451a855877c87188f1235ef663f4350b0ca25fc84ed8f626c287b87b3e729" exitCode=0 Oct 31 19:17:44.672176 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:44.672042 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a86c21-e504-455a-83d5-e68baf8191c3","Type":"ContainerDied","Data":"f4a451a855877c87188f1235ef663f4350b0ca25fc84ed8f626c287b87b3e729"} Oct 31 19:17:44.672176 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:44.672090 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a86c21-e504-455a-83d5-e68baf8191c3","Type":"ContainerStarted","Data":"6234f59a956f776be1d08a605e105d4cda22e5e8f084d29aedbe51fa79392954"} Oct 31 19:17:45.679155 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:45.679124 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a86c21-e504-455a-83d5-e68baf8191c3","Type":"ContainerStarted","Data":"2fd0daaca34fc1e53da53de2963bab2a3954efc80a6aa6a208ef540ba7d037eb"} Oct 31 19:17:45.679555 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:45.679187 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a86c21-e504-455a-83d5-e68baf8191c3","Type":"ContainerStarted","Data":"05f2facf3fdf2548586a3c2fe8bfd7a8d47947959e4164d8c01deeae336f1939"} Oct 31 19:17:45.679555 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:45.679213 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a86c21-e504-455a-83d5-e68baf8191c3","Type":"ContainerStarted","Data":"3b13c96986055d8a988e277c519df38b2c792f0bdac914985b341bf92c1dfeff"} Oct 31 19:17:45.679555 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:45.679243 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a86c21-e504-455a-83d5-e68baf8191c3","Type":"ContainerStarted","Data":"2e1165a7ef5009a395c26e093de5a554ff63fff7af2822985da383133d798030"} Oct 31 19:17:45.679555 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:45.679267 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a86c21-e504-455a-83d5-e68baf8191c3","Type":"ContainerStarted","Data":"ec0ee3b6a3ac63fde03130d3acad8981cfa150f71277e23d630296a66f1d345a"} Oct 31 19:17:45.679555 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:45.679290 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"45a86c21-e504-455a-83d5-e68baf8191c3","Type":"ContainerStarted","Data":"3aaa613b177442116cd4a50837344c86267ccd92a88433198e8afc7c3a5191d3"} Oct 31 19:17:45.796776 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:45.796722 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.783526623 podStartE2EDuration="3.796707946s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.181323ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:45.79560229 +0000 UTC m=+37.428996198" watchObservedRunningTime="2026-10-31 19:17:45.796707946 +0000 UTC m=+37.430101904" Oct 31 19:17:46.408506 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.405965 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 19:17:46.408506 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.406554 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="prometheus" containerID="cri-o://303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b" gracePeriod=600 Oct 31 19:17:46.408506 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.406797 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="kube-rbac-proxy" containerID="cri-o://c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054" gracePeriod=600 Oct 31 19:17:46.408506 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.406936 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="kube-rbac-proxy-web" containerID="cri-o://3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c" gracePeriod=600 Oct 31 19:17:46.408506 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.407033 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="thanos-sidecar" containerID="cri-o://736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3" gracePeriod=600 Oct 31 19:17:46.408506 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.407079 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc" gracePeriod=600 Oct 31 19:17:46.408506 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.407120 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="config-reloader" containerID="cri-o://e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b" gracePeriod=600 Oct 31 19:17:46.648083 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.648060 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:46.687106 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687080 6107 generic.go:356] "Generic (PLEG): container finished" podID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerID="f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc" exitCode=0 Oct 31 19:17:46.687106 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687103 6107 generic.go:356] "Generic (PLEG): container finished" podID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerID="c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054" exitCode=0 Oct 31 19:17:46.687106 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687109 6107 generic.go:356] "Generic (PLEG): container finished" podID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerID="3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c" exitCode=0 Oct 31 19:17:46.687464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687115 6107 generic.go:356] "Generic (PLEG): container finished" podID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerID="736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3" exitCode=0 Oct 31 19:17:46.687464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687120 6107 generic.go:356] "Generic (PLEG): container finished" podID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerID="e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b" exitCode=0 Oct 31 19:17:46.687464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687125 6107 generic.go:356] "Generic (PLEG): container finished" podID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerID="303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b" exitCode=0 Oct 31 19:17:46.687464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687155 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:46.687464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687160 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerDied","Data":"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc"} Oct 31 19:17:46.687464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687233 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerDied","Data":"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054"} Oct 31 19:17:46.687464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687259 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerDied","Data":"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c"} Oct 31 19:17:46.687464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687284 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerDied","Data":"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3"} Oct 31 19:17:46.687464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687308 6107 scope.go:122] "RemoveContainer" containerID="f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc" Oct 31 19:17:46.687464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687313 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerDied","Data":"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b"} Oct 31 19:17:46.687464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687418 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerDied","Data":"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b"} Oct 31 19:17:46.687464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.687442 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"90c88061-9a64-4653-96d8-e40f15c7f60c","Type":"ContainerDied","Data":"8fc52594fbc580ce4ed493bdf7ec0ab5245602e6c62f8c43cfad99ebd61a70b3"} Oct 31 19:17:46.694198 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.694182 6107 scope.go:122] "RemoveContainer" containerID="c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054" Oct 31 19:17:46.700790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.700776 6107 scope.go:122] "RemoveContainer" containerID="3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c" Oct 31 19:17:46.706686 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.706672 6107 scope.go:122] "RemoveContainer" containerID="736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3" Oct 31 19:17:46.712627 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.712611 6107 scope.go:122] "RemoveContainer" containerID="e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b" Oct 31 19:17:46.718624 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.718610 6107 scope.go:122] "RemoveContainer" containerID="303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b" Oct 31 19:17:46.724848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.724835 6107 scope.go:122] "RemoveContainer" containerID="12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58" Oct 31 19:17:46.725216 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725202 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-trusted-ca-bundle\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725288 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725228 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-metrics-client-certs\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725288 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725254 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-config-out\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725369 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725283 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-2\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725369 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725320 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-db\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725369 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725348 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-0\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725408 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-kubelet-serving-ca-bundle\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725437 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-grpc-tls\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725473 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-tls-assets\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-tls-assets\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725510 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-web-config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-web-config\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725723 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725541 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-tls\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725723 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725567 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-thanos-prometheus-http-client-file\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725723 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725608 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-serving-certs-ca-bundle\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725723 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725649 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-kube-api-access-vj4gh\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-kube-api-access-vj4gh\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725723 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725701 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725996 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725727 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-metrics-client-ca\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725996 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725755 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-kube-rbac-proxy\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725996 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725799 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-config\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725996 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725843 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.725996 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725636 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-trusted-ca-bundle" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:46.726290 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.725990 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-2" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:46.726341 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.726290 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-1\") pod \"90c88061-9a64-4653-96d8-e40f15c7f60c\" (UID: \"90c88061-9a64-4653-96d8-e40f15c7f60c\") " Oct 31 19:17:46.726466 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.726386 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-metrics-client-ca" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:46.726624 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.726605 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-kubelet-serving-ca-bundle" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:46.726624 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.726616 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-db" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:17:46.726765 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.726671 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-1" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:46.728013 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.727866 6107 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-metrics-client-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.728013 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.727916 6107 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.728013 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.727932 6107 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.728013 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.727947 6107 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.728013 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.727963 6107 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-db\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.728013 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.727980 6107 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.728371 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.728041 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-serving-certs-ca-bundle" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:46.728433 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.728393 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-0" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:46.729126 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.729102 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-thanos-sidecar-tls" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:46.729249 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.729212 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-kube-rbac-proxy-web" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:46.729688 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.729626 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-thanos-prometheus-http-client-file" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:46.729908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.729861 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-tls-assets" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:17:46.730098 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.730061 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-metrics-client-certs" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:46.730098 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.730085 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-kube-api-access-vj4gh" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "kube-api-access-vj4gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:17:46.730222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.730163 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-config-out" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:17:46.730564 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.730543 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-kube-rbac-proxy" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:46.730953 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.730932 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-config" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:46.731469 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.731453 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-tls" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:46.731923 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.731901 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-grpc-tls" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:46.742236 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.742212 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-web-config" pod "90c88061-9a64-4653-96d8-e40f15c7f60c" (UID: "90c88061-9a64-4653-96d8-e40f15c7f60c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:46.750912 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.750873 6107 scope.go:122] "RemoveContainer" containerID="f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc" Oct 31 19:17:46.751223 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:46.751202 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": container with ID starting with f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc not found: ID does not exist" containerID="f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc" Oct 31 19:17:46.751283 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.751234 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc"} err="failed to get container status \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": rpc error: code = NotFound desc = could not find container \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": container with ID starting with f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc not found: ID does not exist" Oct 31 19:17:46.751283 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.751252 6107 scope.go:122] "RemoveContainer" containerID="c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054" Oct 31 19:17:46.751548 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:46.751532 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": container with ID starting with c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054 not found: ID does not exist" containerID="c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054" Oct 31 19:17:46.751591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.751555 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054"} err="failed to get container status \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": rpc error: code = NotFound desc = could not find container \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": container with ID starting with c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054 not found: ID does not exist" Oct 31 19:17:46.751591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.751570 6107 scope.go:122] "RemoveContainer" containerID="3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c" Oct 31 19:17:46.751834 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:46.751815 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": container with ID starting with 3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c not found: ID does not exist" containerID="3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c" Oct 31 19:17:46.751897 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.751839 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c"} err="failed to get container status \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": rpc error: code = NotFound desc = could not find container \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": container with ID starting with 3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c not found: ID does not exist" Oct 31 19:17:46.751897 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.751856 6107 scope.go:122] "RemoveContainer" containerID="736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3" Oct 31 19:17:46.752123 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:46.752104 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": container with ID starting with 736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3 not found: ID does not exist" containerID="736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3" Oct 31 19:17:46.752174 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.752129 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3"} err="failed to get container status \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": rpc error: code = NotFound desc = could not find container \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": container with ID starting with 736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3 not found: ID does not exist" Oct 31 19:17:46.752174 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.752146 6107 scope.go:122] "RemoveContainer" containerID="e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b" Oct 31 19:17:46.752368 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:46.752355 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": container with ID starting with e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b not found: ID does not exist" containerID="e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b" Oct 31 19:17:46.752407 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.752370 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b"} err="failed to get container status \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": rpc error: code = NotFound desc = could not find container \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": container with ID starting with e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b not found: ID does not exist" Oct 31 19:17:46.752407 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.752382 6107 scope.go:122] "RemoveContainer" containerID="303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b" Oct 31 19:17:46.752612 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:46.752595 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": container with ID starting with 303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b not found: ID does not exist" containerID="303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b" Oct 31 19:17:46.752654 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.752617 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b"} err="failed to get container status \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": rpc error: code = NotFound desc = could not find container \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": container with ID starting with 303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b not found: ID does not exist" Oct 31 19:17:46.752654 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.752634 6107 scope.go:122] "RemoveContainer" containerID="12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58" Oct 31 19:17:46.752866 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:46.752849 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": container with ID starting with 12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58 not found: ID does not exist" containerID="12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58" Oct 31 19:17:46.752999 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.752870 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58"} err="failed to get container status \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": rpc error: code = NotFound desc = could not find container \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": container with ID starting with 12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58 not found: ID does not exist" Oct 31 19:17:46.752999 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.752905 6107 scope.go:122] "RemoveContainer" containerID="f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc" Oct 31 19:17:46.753114 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.753096 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc"} err="failed to get container status \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": rpc error: code = NotFound desc = could not find container \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": container with ID starting with f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc not found: ID does not exist" Oct 31 19:17:46.753156 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.753114 6107 scope.go:122] "RemoveContainer" containerID="c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054" Oct 31 19:17:46.753312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.753297 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054"} err="failed to get container status \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": rpc error: code = NotFound desc = could not find container \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": container with ID starting with c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054 not found: ID does not exist" Oct 31 19:17:46.753312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.753312 6107 scope.go:122] "RemoveContainer" containerID="3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c" Oct 31 19:17:46.753540 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.753522 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c"} err="failed to get container status \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": rpc error: code = NotFound desc = could not find container \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": container with ID starting with 3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c not found: ID does not exist" Oct 31 19:17:46.753581 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.753541 6107 scope.go:122] "RemoveContainer" containerID="736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3" Oct 31 19:17:46.753758 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.753742 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3"} err="failed to get container status \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": rpc error: code = NotFound desc = could not find container \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": container with ID starting with 736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3 not found: ID does not exist" Oct 31 19:17:46.753758 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.753757 6107 scope.go:122] "RemoveContainer" containerID="e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b" Oct 31 19:17:46.753988 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.753973 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b"} err="failed to get container status \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": rpc error: code = NotFound desc = could not find container \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": container with ID starting with e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b not found: ID does not exist" Oct 31 19:17:46.754049 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.753987 6107 scope.go:122] "RemoveContainer" containerID="303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b" Oct 31 19:17:46.754193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.754178 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b"} err="failed to get container status \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": rpc error: code = NotFound desc = could not find container \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": container with ID starting with 303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b not found: ID does not exist" Oct 31 19:17:46.754248 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.754193 6107 scope.go:122] "RemoveContainer" containerID="12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58" Oct 31 19:17:46.754417 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.754400 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58"} err="failed to get container status \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": rpc error: code = NotFound desc = could not find container \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": container with ID starting with 12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58 not found: ID does not exist" Oct 31 19:17:46.754461 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.754419 6107 scope.go:122] "RemoveContainer" containerID="f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc" Oct 31 19:17:46.754638 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.754616 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc"} err="failed to get container status \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": rpc error: code = NotFound desc = could not find container \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": container with ID starting with f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc not found: ID does not exist" Oct 31 19:17:46.754638 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.754637 6107 scope.go:122] "RemoveContainer" containerID="c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054" Oct 31 19:17:46.754849 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.754834 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054"} err="failed to get container status \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": rpc error: code = NotFound desc = could not find container \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": container with ID starting with c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054 not found: ID does not exist" Oct 31 19:17:46.754849 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.754848 6107 scope.go:122] "RemoveContainer" containerID="3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c" Oct 31 19:17:46.755116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.755100 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c"} err="failed to get container status \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": rpc error: code = NotFound desc = could not find container \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": container with ID starting with 3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c not found: ID does not exist" Oct 31 19:17:46.755165 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.755128 6107 scope.go:122] "RemoveContainer" containerID="736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3" Oct 31 19:17:46.755333 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.755315 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3"} err="failed to get container status \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": rpc error: code = NotFound desc = could not find container \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": container with ID starting with 736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3 not found: ID does not exist" Oct 31 19:17:46.755403 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.755335 6107 scope.go:122] "RemoveContainer" containerID="e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b" Oct 31 19:17:46.755552 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.755535 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b"} err="failed to get container status \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": rpc error: code = NotFound desc = could not find container \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": container with ID starting with e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b not found: ID does not exist" Oct 31 19:17:46.755618 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.755553 6107 scope.go:122] "RemoveContainer" containerID="303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b" Oct 31 19:17:46.755792 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.755776 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b"} err="failed to get container status \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": rpc error: code = NotFound desc = could not find container \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": container with ID starting with 303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b not found: ID does not exist" Oct 31 19:17:46.755858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.755793 6107 scope.go:122] "RemoveContainer" containerID="12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58" Oct 31 19:17:46.756052 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.756034 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58"} err="failed to get container status \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": rpc error: code = NotFound desc = could not find container \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": container with ID starting with 12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58 not found: ID does not exist" Oct 31 19:17:46.756108 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.756053 6107 scope.go:122] "RemoveContainer" containerID="f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc" Oct 31 19:17:46.756257 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.756237 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc"} err="failed to get container status \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": rpc error: code = NotFound desc = could not find container \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": container with ID starting with f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc not found: ID does not exist" Oct 31 19:17:46.756326 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.756258 6107 scope.go:122] "RemoveContainer" containerID="c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054" Oct 31 19:17:46.756470 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.756456 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054"} err="failed to get container status \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": rpc error: code = NotFound desc = could not find container \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": container with ID starting with c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054 not found: ID does not exist" Oct 31 19:17:46.756522 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.756470 6107 scope.go:122] "RemoveContainer" containerID="3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c" Oct 31 19:17:46.756676 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.756656 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c"} err="failed to get container status \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": rpc error: code = NotFound desc = could not find container \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": container with ID starting with 3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c not found: ID does not exist" Oct 31 19:17:46.756716 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.756683 6107 scope.go:122] "RemoveContainer" containerID="736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3" Oct 31 19:17:46.756923 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.756905 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3"} err="failed to get container status \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": rpc error: code = NotFound desc = could not find container \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": container with ID starting with 736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3 not found: ID does not exist" Oct 31 19:17:46.757009 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.756924 6107 scope.go:122] "RemoveContainer" containerID="e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b" Oct 31 19:17:46.757115 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.757096 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b"} err="failed to get container status \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": rpc error: code = NotFound desc = could not find container \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": container with ID starting with e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b not found: ID does not exist" Oct 31 19:17:46.757159 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.757116 6107 scope.go:122] "RemoveContainer" containerID="303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b" Oct 31 19:17:46.757332 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.757315 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b"} err="failed to get container status \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": rpc error: code = NotFound desc = could not find container \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": container with ID starting with 303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b not found: ID does not exist" Oct 31 19:17:46.757371 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.757333 6107 scope.go:122] "RemoveContainer" containerID="12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58" Oct 31 19:17:46.757544 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.757529 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58"} err="failed to get container status \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": rpc error: code = NotFound desc = could not find container \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": container with ID starting with 12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58 not found: ID does not exist" Oct 31 19:17:46.757544 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.757543 6107 scope.go:122] "RemoveContainer" containerID="f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc" Oct 31 19:17:46.757751 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.757736 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc"} err="failed to get container status \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": rpc error: code = NotFound desc = could not find container \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": container with ID starting with f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc not found: ID does not exist" Oct 31 19:17:46.757801 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.757750 6107 scope.go:122] "RemoveContainer" containerID="c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054" Oct 31 19:17:46.757955 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.757935 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054"} err="failed to get container status \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": rpc error: code = NotFound desc = could not find container \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": container with ID starting with c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054 not found: ID does not exist" Oct 31 19:17:46.758027 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.757955 6107 scope.go:122] "RemoveContainer" containerID="3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c" Oct 31 19:17:46.758169 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.758150 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c"} err="failed to get container status \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": rpc error: code = NotFound desc = could not find container \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": container with ID starting with 3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c not found: ID does not exist" Oct 31 19:17:46.758208 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.758169 6107 scope.go:122] "RemoveContainer" containerID="736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3" Oct 31 19:17:46.758366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.758348 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3"} err="failed to get container status \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": rpc error: code = NotFound desc = could not find container \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": container with ID starting with 736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3 not found: ID does not exist" Oct 31 19:17:46.758432 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.758366 6107 scope.go:122] "RemoveContainer" containerID="e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b" Oct 31 19:17:46.758605 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.758587 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b"} err="failed to get container status \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": rpc error: code = NotFound desc = could not find container \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": container with ID starting with e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b not found: ID does not exist" Oct 31 19:17:46.758651 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.758605 6107 scope.go:122] "RemoveContainer" containerID="303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b" Oct 31 19:17:46.758828 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.758812 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b"} err="failed to get container status \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": rpc error: code = NotFound desc = could not find container \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": container with ID starting with 303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b not found: ID does not exist" Oct 31 19:17:46.758871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.758828 6107 scope.go:122] "RemoveContainer" containerID="12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58" Oct 31 19:17:46.759070 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.759046 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58"} err="failed to get container status \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": rpc error: code = NotFound desc = could not find container \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": container with ID starting with 12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58 not found: ID does not exist" Oct 31 19:17:46.759113 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.759070 6107 scope.go:122] "RemoveContainer" containerID="f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc" Oct 31 19:17:46.759324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.759308 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc"} err="failed to get container status \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": rpc error: code = NotFound desc = could not find container \"f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc\": container with ID starting with f3088a587601df3e5005f6d067dc396212fd8543d40d7e8e1328c056583e45bc not found: ID does not exist" Oct 31 19:17:46.759377 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.759325 6107 scope.go:122] "RemoveContainer" containerID="c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054" Oct 31 19:17:46.759522 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.759503 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054"} err="failed to get container status \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": rpc error: code = NotFound desc = could not find container \"c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054\": container with ID starting with c01684482eeccf71ed2ba8006d72b354cd5d221177e2935329e2239bf20a2054 not found: ID does not exist" Oct 31 19:17:46.759588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.759523 6107 scope.go:122] "RemoveContainer" containerID="3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c" Oct 31 19:17:46.759749 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.759734 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c"} err="failed to get container status \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": rpc error: code = NotFound desc = could not find container \"3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c\": container with ID starting with 3226cafe49da1345a8cb36f77bc910fff3472f6633c1250e2cbf6ce12295e89c not found: ID does not exist" Oct 31 19:17:46.759749 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.759748 6107 scope.go:122] "RemoveContainer" containerID="736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3" Oct 31 19:17:46.759960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.759941 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3"} err="failed to get container status \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": rpc error: code = NotFound desc = could not find container \"736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3\": container with ID starting with 736029656db6c9dde49618f397b7d00953dae38ee8339c0d5e63af1a8d3091e3 not found: ID does not exist" Oct 31 19:17:46.760006 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.759961 6107 scope.go:122] "RemoveContainer" containerID="e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b" Oct 31 19:17:46.760182 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.760161 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b"} err="failed to get container status \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": rpc error: code = NotFound desc = could not find container \"e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b\": container with ID starting with e1c44b8e085a32081d8a9d79ee8d939a2e1bdb008f2b8a2304e059c9782f900b not found: ID does not exist" Oct 31 19:17:46.760229 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.760183 6107 scope.go:122] "RemoveContainer" containerID="303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b" Oct 31 19:17:46.760383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.760356 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b"} err="failed to get container status \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": rpc error: code = NotFound desc = could not find container \"303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b\": container with ID starting with 303ec19ec01c19519b543f202c320eab54ecb8bc4d1635bf2ebc572e5247310b not found: ID does not exist" Oct 31 19:17:46.760383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.760379 6107 scope.go:122] "RemoveContainer" containerID="12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58" Oct 31 19:17:46.760588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.760571 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58"} err="failed to get container status \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": rpc error: code = NotFound desc = could not find container \"12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58\": container with ID starting with 12988abf1afd67aa0574c8380b0474390c0be95e2d80a90af56cad6d4c02db58 not found: ID does not exist" Oct 31 19:17:46.829616 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829588 6107 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-web-config\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829616 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829613 6107 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829616 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829625 6107 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829646 6107 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829657 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj4gh\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-kube-api-access-vj4gh\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829666 6107 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829682 6107 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-kube-rbac-proxy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829691 6107 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-config\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829700 6107 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829708 6107 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-metrics-client-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829717 6107 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/90c88061-9a64-4653-96d8-e40f15c7f60c-config-out\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829725 6107 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/90c88061-9a64-4653-96d8-e40f15c7f60c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829734 6107 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90c88061-9a64-4653-96d8-e40f15c7f60c-secret-grpc-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:46.829837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:46.829742 6107 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/90c88061-9a64-4653-96d8-e40f15c7f60c-tls-assets\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:47.100525 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.100487 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 19:17:47.108893 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.108859 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 19:17:47.135863 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.135836 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 19:17:47.136131 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136119 6107 cpu_manager.go:414] "containerMap: removing container" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="kube-rbac-proxy-web" Oct 31 19:17:47.136181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136132 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="kube-rbac-proxy-web" Oct 31 19:17:47.136181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136140 6107 cpu_manager.go:414] "containerMap: removing container" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="thanos-sidecar" Oct 31 19:17:47.136181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136145 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="thanos-sidecar" Oct 31 19:17:47.136181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136156 6107 cpu_manager.go:414] "containerMap: removing container" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="kube-rbac-proxy" Oct 31 19:17:47.136181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136162 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="kube-rbac-proxy" Oct 31 19:17:47.136181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136169 6107 cpu_manager.go:414] "containerMap: removing container" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="init-config-reloader" Oct 31 19:17:47.136181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136174 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="init-config-reloader" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136187 6107 cpu_manager.go:414] "containerMap: removing container" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="kube-rbac-proxy-thanos" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136192 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="kube-rbac-proxy-thanos" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136203 6107 cpu_manager.go:414] "containerMap: removing container" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="config-reloader" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136207 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="config-reloader" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136218 6107 cpu_manager.go:414] "containerMap: removing container" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="prometheus" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136223 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="prometheus" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136257 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="kube-rbac-proxy-web" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136264 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="thanos-sidecar" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136269 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="kube-rbac-proxy" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136275 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="init-config-reloader" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136282 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="kube-rbac-proxy-thanos" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136288 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="config-reloader" Oct 31 19:17:47.136396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.136294 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" containerName="prometheus" Oct 31 19:17:47.140032 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.140015 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.194038 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194000 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 19:17:47.194267 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194247 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 19:17:47.194346 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194282 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 19:17:47.194346 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194318 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 19:17:47.194437 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194350 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 19:17:47.194437 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194352 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 19:17:47.194437 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194386 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 19:17:47.194437 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194387 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 19:17:47.194599 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194499 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 19:17:47.194737 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194721 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c47hb7itrpuc4\"" Oct 31 19:17:47.194803 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194787 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 19:17:47.194858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194842 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 19:17:47.194939 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.194926 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 19:17:47.195048 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.195035 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qq6mx\"" Oct 31 19:17:47.195119 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.195107 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 19:17:47.233820 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.233787 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwsf2\" (UniqueName: \"kubernetes.io/projected/79735aa7-bcbb-4428-8082-1c41a0db7b46-kube-api-access-xwsf2\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.233989 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.233831 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.233989 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.233853 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234071 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.233992 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79735aa7-bcbb-4428-8082-1c41a0db7b46-config-out\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234071 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234020 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234071 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234035 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-config\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234187 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234075 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234187 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234120 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-web-config\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234187 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234166 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234314 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234258 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79735aa7-bcbb-4428-8082-1c41a0db7b46-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234314 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234275 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234314 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234294 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234446 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234313 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234446 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234329 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234446 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234409 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234446 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234435 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234614 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234474 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234614 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234489 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234614 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234582 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.234614 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.234609 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.240722 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.240704 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 19:17:47.240799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.240764 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 19:17:47.260502 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.260447 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 19:17:47.335646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.335608 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.335646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.335649 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.335915 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.335690 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.335915 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.335712 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.335915 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.335737 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwsf2\" (UniqueName: \"kubernetes.io/projected/79735aa7-bcbb-4428-8082-1c41a0db7b46-kube-api-access-xwsf2\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.335915 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.335818 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.335915 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.335848 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.335915 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.335875 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79735aa7-bcbb-4428-8082-1c41a0db7b46-config-out\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.335918 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.335942 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-config\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.335966 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.335993 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-web-config\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336015 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336052 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79735aa7-bcbb-4428-8082-1c41a0db7b46-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336085 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336121 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336153 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336178 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336212 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336235 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.336394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336307 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.337068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336478 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.337068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336550 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.337068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336734 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.337068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.336866 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.338927 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.338789 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79735aa7-bcbb-4428-8082-1c41a0db7b46-config-out\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.338927 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.338856 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.339095 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.338934 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.339095 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.338973 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.339095 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.338972 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.339586 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.339272 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.339702 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.339589 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.340078 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.340053 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.340786 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.340761 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79735aa7-bcbb-4428-8082-1c41a0db7b46-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.341084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.341043 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-web-config\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.341489 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.341457 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-config\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.341594 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.341577 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.341820 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.341802 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79735aa7-bcbb-4428-8082-1c41a0db7b46-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.342085 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.342065 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79735aa7-bcbb-4428-8082-1c41a0db7b46-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.390116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.390086 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwsf2\" (UniqueName: \"kubernetes.io/projected/79735aa7-bcbb-4428-8082-1c41a0db7b46-kube-api-access-xwsf2\") pod \"prometheus-k8s-0\" (UID: \"79735aa7-bcbb-4428-8082-1c41a0db7b46\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.458426 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.458395 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:17:47.877942 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:47.877911 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 19:17:47.881315 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:17:47.881289 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79735aa7_bcbb_4428_8082_1c41a0db7b46.slice/crio-7d9d42a335372a8562877d2aa682bc6ed738552371e5f514e5565cec78ef2bd9 WatchSource:0}: Error finding container 7d9d42a335372a8562877d2aa682bc6ed738552371e5f514e5565cec78ef2bd9: Status 404 returned error can't find the container with id 7d9d42a335372a8562877d2aa682bc6ed738552371e5f514e5565cec78ef2bd9 Oct 31 19:17:48.484097 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:48.484056 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c88061-9a64-4653-96d8-e40f15c7f60c" path="/var/lib/kubelet/pods/90c88061-9a64-4653-96d8-e40f15c7f60c/volumes" Oct 31 19:17:48.696315 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:48.696282 6107 generic.go:356] "Generic (PLEG): container finished" podID="79735aa7-bcbb-4428-8082-1c41a0db7b46" containerID="3aabdd54c1a53eb62d906a2f4516c58c738c3e11d0778e897b05664b14c61672" exitCode=0 Oct 31 19:17:48.696441 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:48.696334 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79735aa7-bcbb-4428-8082-1c41a0db7b46","Type":"ContainerDied","Data":"3aabdd54c1a53eb62d906a2f4516c58c738c3e11d0778e897b05664b14c61672"} Oct 31 19:17:48.696441 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:48.696425 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79735aa7-bcbb-4428-8082-1c41a0db7b46","Type":"ContainerStarted","Data":"7d9d42a335372a8562877d2aa682bc6ed738552371e5f514e5565cec78ef2bd9"} Oct 31 19:17:49.703649 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:49.703613 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79735aa7-bcbb-4428-8082-1c41a0db7b46","Type":"ContainerStarted","Data":"7fcf7c8356ff9f7c6fa5a0774de672a62ea4c1d706733287ab7183c1add977ce"} Oct 31 19:17:49.704063 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:49.703681 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79735aa7-bcbb-4428-8082-1c41a0db7b46","Type":"ContainerStarted","Data":"ecf41341af1cc39aaf57311af5da8cfdcaa079a0f075e8035e13a72a7e5950be"} Oct 31 19:17:49.704063 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:49.703722 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79735aa7-bcbb-4428-8082-1c41a0db7b46","Type":"ContainerStarted","Data":"7ee01680e31511a8c25ee6baeb3da46e29367ad90cb0fcbad136e317d6ce4260"} Oct 31 19:17:49.704063 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:49.703750 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79735aa7-bcbb-4428-8082-1c41a0db7b46","Type":"ContainerStarted","Data":"2bfb5162883d3a061825fdd9716a182289f601f9009b5d9b32d2d4861c906906"} Oct 31 19:17:49.704063 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:49.703772 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79735aa7-bcbb-4428-8082-1c41a0db7b46","Type":"ContainerStarted","Data":"425b097871413f186105342058d1cf71b32926257c914d3b7730f2a7a3f8ac62"} Oct 31 19:17:49.704063 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:49.703806 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79735aa7-bcbb-4428-8082-1c41a0db7b46","Type":"ContainerStarted","Data":"0df1321bd56a249f65fd96753f86e12a6198f0b7ce8390b6e8580889a23a5c9d"} Oct 31 19:17:49.829468 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:49.829407 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.812522709 podStartE2EDuration="2.829386501s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.863792ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:49.829339665 +0000 UTC m=+41.462733573" watchObservedRunningTime="2026-10-31 19:17:49.829386501 +0000 UTC m=+41.462780409" Oct 31 19:17:50.401446 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:17:50.401407 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:17:52.458630 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:17:52.458586 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:18:20.403769 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:18:20.403734 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:18:43.090681 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:18:43.090646 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:18:43.105421 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:18:43.105396 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:18:43.900316 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:18:43.900291 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 19:18:50.401484 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:18:50.401402 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:19:20.400942 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:19:20.400901 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:19:50.401336 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:19:50.401299 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:20:20.404550 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:20:20.404467 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:20:50.401487 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:20:50.401450 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:21:16.817375 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:16.817342 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc"] Oct 31 19:21:16.820752 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:16.820736 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:16.907255 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:16.907221 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnjv\" (UniqueName: \"kubernetes.io/projected/37f99e10-708a-40c3-bddd-19d13f813cb7-kube-api-access-tdnjv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc\" (UID: \"37f99e10-708a-40c3-bddd-19d13f813cb7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:16.907255 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:16.907258 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc\" (UID: \"37f99e10-708a-40c3-bddd-19d13f813cb7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:16.907457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:16.907340 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc\" (UID: \"37f99e10-708a-40c3-bddd-19d13f813cb7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:16.926201 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:16.926176 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 19:21:16.926201 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:16.926181 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 19:21:16.926201 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:16.926213 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-szgbd\"" Oct 31 19:21:16.926824 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:16.926807 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc"] Oct 31 19:21:17.007937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:17.007902 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnjv\" (UniqueName: \"kubernetes.io/projected/37f99e10-708a-40c3-bddd-19d13f813cb7-kube-api-access-tdnjv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc\" (UID: \"37f99e10-708a-40c3-bddd-19d13f813cb7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:17.007937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:17.007936 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc\" (UID: \"37f99e10-708a-40c3-bddd-19d13f813cb7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:17.008179 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:17.007969 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc\" (UID: \"37f99e10-708a-40c3-bddd-19d13f813cb7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:17.008307 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:17.008286 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc\" (UID: \"37f99e10-708a-40c3-bddd-19d13f813cb7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:17.008365 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:17.008311 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc\" (UID: \"37f99e10-708a-40c3-bddd-19d13f813cb7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:17.061655 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:17.061627 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnjv\" (UniqueName: \"kubernetes.io/projected/37f99e10-708a-40c3-bddd-19d13f813cb7-kube-api-access-tdnjv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc\" (UID: \"37f99e10-708a-40c3-bddd-19d13f813cb7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:17.135791 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:17.135711 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:17.494181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:17.494157 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc"] Oct 31 19:21:17.497261 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:21:17.497235 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f99e10_708a_40c3_bddd_19d13f813cb7.slice/crio-059fdda7b73a98931908357829b065944eca350a931c2b1e685869062f30e4f1 WatchSource:0}: Error finding container 059fdda7b73a98931908357829b065944eca350a931c2b1e685869062f30e4f1: Status 404 returned error can't find the container with id 059fdda7b73a98931908357829b065944eca350a931c2b1e685869062f30e4f1 Oct 31 19:21:17.499552 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:17.499534 6107 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:21:18.083008 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.082975 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt"] Oct 31 19:21:18.086075 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.086056 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:18.115830 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.115805 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr467\" (UniqueName: \"kubernetes.io/projected/4d212d46-98dd-4b4c-a898-629d409f548e-kube-api-access-jr467\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt\" (UID: \"4d212d46-98dd-4b4c-a898-629d409f548e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:18.115956 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.115856 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt\" (UID: \"4d212d46-98dd-4b4c-a898-629d409f548e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:18.116014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.115984 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt\" (UID: \"4d212d46-98dd-4b4c-a898-629d409f548e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:18.180427 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.180395 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-98wrm\"" Oct 31 19:21:18.181229 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.181210 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt"] Oct 31 19:21:18.182674 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.182652 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v"] Oct 31 19:21:18.185778 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.185761 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:18.217063 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.217042 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v\" (UID: \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:18.217168 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.217082 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt\" (UID: \"4d212d46-98dd-4b4c-a898-629d409f548e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:18.217168 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.217153 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt\" (UID: \"4d212d46-98dd-4b4c-a898-629d409f548e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:18.217237 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.217184 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v\" (UID: \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:18.217237 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.217220 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmg9\" (UniqueName: \"kubernetes.io/projected/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-kube-api-access-jxmg9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v\" (UID: \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:18.217319 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.217266 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr467\" (UniqueName: \"kubernetes.io/projected/4d212d46-98dd-4b4c-a898-629d409f548e-kube-api-access-jr467\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt\" (UID: \"4d212d46-98dd-4b4c-a898-629d409f548e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:18.217477 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.217460 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt\" (UID: \"4d212d46-98dd-4b4c-a898-629d409f548e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:18.217477 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.217473 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt\" (UID: \"4d212d46-98dd-4b4c-a898-629d409f548e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:18.237245 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.237218 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-lkxw8\"" Oct 31 19:21:18.244982 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.244958 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v"] Oct 31 19:21:18.284992 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.284965 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr467\" (UniqueName: \"kubernetes.io/projected/4d212d46-98dd-4b4c-a898-629d409f548e-kube-api-access-jr467\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt\" (UID: \"4d212d46-98dd-4b4c-a898-629d409f548e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:18.285830 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.285810 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg"] Oct 31 19:21:18.289687 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.289670 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:18.318166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.318138 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmg9\" (UniqueName: \"kubernetes.io/projected/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-kube-api-access-jxmg9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v\" (UID: \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:18.318242 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.318178 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg\" (UID: \"15380b16-b44d-4c07-972f-30afada962ce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:18.318242 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.318218 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v\" (UID: \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:18.318324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.318241 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg\" (UID: \"15380b16-b44d-4c07-972f-30afada962ce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:18.318324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.318265 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmkr7\" (UniqueName: \"kubernetes.io/projected/15380b16-b44d-4c07-972f-30afada962ce-kube-api-access-bmkr7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg\" (UID: \"15380b16-b44d-4c07-972f-30afada962ce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:18.318409 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.318392 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v\" (UID: \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:18.318548 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.318531 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v\" (UID: \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:18.318662 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.318647 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v\" (UID: \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:18.337908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.337843 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-nv2tg\"" Oct 31 19:21:18.343065 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.343042 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg"] Oct 31 19:21:18.370947 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.370915 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmg9\" (UniqueName: \"kubernetes.io/projected/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-kube-api-access-jxmg9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v\" (UID: \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:18.384535 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.384508 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" event={"ID":"37f99e10-708a-40c3-bddd-19d13f813cb7","Type":"ContainerStarted","Data":"059fdda7b73a98931908357829b065944eca350a931c2b1e685869062f30e4f1"} Oct 31 19:21:18.386312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.386288 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45"] Oct 31 19:21:18.389815 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.389798 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:18.401315 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.401297 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:18.419437 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.419413 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg\" (UID: \"15380b16-b44d-4c07-972f-30afada962ce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:18.419521 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.419453 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmkr7\" (UniqueName: \"kubernetes.io/projected/15380b16-b44d-4c07-972f-30afada962ce-kube-api-access-bmkr7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg\" (UID: \"15380b16-b44d-4c07-972f-30afada962ce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:18.419521 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.419497 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45\" (UID: \"b269313c-091e-4d9c-a229-5c7f1e71ea30\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:18.419626 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.419550 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqvwj\" (UniqueName: \"kubernetes.io/projected/b269313c-091e-4d9c-a229-5c7f1e71ea30-kube-api-access-wqvwj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45\" (UID: \"b269313c-091e-4d9c-a229-5c7f1e71ea30\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:18.419692 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.419670 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45\" (UID: \"b269313c-091e-4d9c-a229-5c7f1e71ea30\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:18.419750 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.419734 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg\" (UID: \"15380b16-b44d-4c07-972f-30afada962ce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:18.419802 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.419770 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg\" (UID: \"15380b16-b44d-4c07-972f-30afada962ce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:18.420068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.420030 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg\" (UID: \"15380b16-b44d-4c07-972f-30afada962ce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:18.438215 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.438162 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-7v2t5\"" Oct 31 19:21:18.443044 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.443018 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45"] Oct 31 19:21:18.472858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.472826 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmkr7\" (UniqueName: \"kubernetes.io/projected/15380b16-b44d-4c07-972f-30afada962ce-kube-api-access-bmkr7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg\" (UID: \"15380b16-b44d-4c07-972f-30afada962ce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:18.501237 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.501216 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:18.520525 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.520503 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45\" (UID: \"b269313c-091e-4d9c-a229-5c7f1e71ea30\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:18.520607 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.520549 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqvwj\" (UniqueName: \"kubernetes.io/projected/b269313c-091e-4d9c-a229-5c7f1e71ea30-kube-api-access-wqvwj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45\" (UID: \"b269313c-091e-4d9c-a229-5c7f1e71ea30\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:18.520607 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.520591 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45\" (UID: \"b269313c-091e-4d9c-a229-5c7f1e71ea30\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:18.520929 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.520910 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45\" (UID: \"b269313c-091e-4d9c-a229-5c7f1e71ea30\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:18.520979 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.520935 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45\" (UID: \"b269313c-091e-4d9c-a229-5c7f1e71ea30\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:18.574320 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.574293 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqvwj\" (UniqueName: \"kubernetes.io/projected/b269313c-091e-4d9c-a229-5c7f1e71ea30-kube-api-access-wqvwj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45\" (UID: \"b269313c-091e-4d9c-a229-5c7f1e71ea30\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:18.605850 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.605796 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:18.705163 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.705129 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:18.772534 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.772505 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt"] Oct 31 19:21:18.859526 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.859452 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v"] Oct 31 19:21:18.862717 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:21:18.862685 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9eb5b7_1463_4dba_9b7a_f4f73052abc7.slice/crio-d1685e7e229a4f64e7fdd8cc210b0ba96baffef224f7a80fcca2f8a42774ea04 WatchSource:0}: Error finding container d1685e7e229a4f64e7fdd8cc210b0ba96baffef224f7a80fcca2f8a42774ea04: Status 404 returned error can't find the container with id d1685e7e229a4f64e7fdd8cc210b0ba96baffef224f7a80fcca2f8a42774ea04 Oct 31 19:21:18.971575 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:18.971550 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg"] Oct 31 19:21:18.974396 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:21:18.974348 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15380b16_b44d_4c07_972f_30afada962ce.slice/crio-2faebe344d4200f9a211183c10130f7bd39a3e26dc76df0fc20f927856256197 WatchSource:0}: Error finding container 2faebe344d4200f9a211183c10130f7bd39a3e26dc76df0fc20f927856256197: Status 404 returned error can't find the container with id 2faebe344d4200f9a211183c10130f7bd39a3e26dc76df0fc20f927856256197 Oct 31 19:21:19.070132 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:19.070097 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45"] Oct 31 19:21:19.073264 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:21:19.073230 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb269313c_091e_4d9c_a229_5c7f1e71ea30.slice/crio-80878711278e1a63c0d06912c306b30b3dc8c688deb206dda025b8b69c3f06c7 WatchSource:0}: Error finding container 80878711278e1a63c0d06912c306b30b3dc8c688deb206dda025b8b69c3f06c7: Status 404 returned error can't find the container with id 80878711278e1a63c0d06912c306b30b3dc8c688deb206dda025b8b69c3f06c7 Oct 31 19:21:19.389869 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:19.389782 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" event={"ID":"b269313c-091e-4d9c-a229-5c7f1e71ea30","Type":"ContainerStarted","Data":"80878711278e1a63c0d06912c306b30b3dc8c688deb206dda025b8b69c3f06c7"} Oct 31 19:21:19.391030 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:19.390996 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" event={"ID":"15380b16-b44d-4c07-972f-30afada962ce","Type":"ContainerStarted","Data":"2faebe344d4200f9a211183c10130f7bd39a3e26dc76df0fc20f927856256197"} Oct 31 19:21:19.392438 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:19.392410 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" event={"ID":"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7","Type":"ContainerStarted","Data":"d1685e7e229a4f64e7fdd8cc210b0ba96baffef224f7a80fcca2f8a42774ea04"} Oct 31 19:21:19.393528 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:19.393505 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" event={"ID":"4d212d46-98dd-4b4c-a898-629d409f548e","Type":"ContainerStarted","Data":"4458e928a7201a7f60b257444febf9eafd9be93ccc4fe3de5b9ebd2195d34085"} Oct 31 19:21:20.305297 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.305261 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2"] Oct 31 19:21:20.309699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.309146 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:20.361754 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.361718 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2"] Oct 31 19:21:20.406514 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.406477 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-mltsq\"" Oct 31 19:21:20.436867 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.436834 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2\" (UID: \"80975e70-dbdf-4a79-a968-c521ef9dea84\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:20.437056 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.436877 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2\" (UID: \"80975e70-dbdf-4a79-a968-c521ef9dea84\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:20.437105 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.437078 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdm28\" (UniqueName: \"kubernetes.io/projected/80975e70-dbdf-4a79-a968-c521ef9dea84-kube-api-access-kdm28\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2\" (UID: \"80975e70-dbdf-4a79-a968-c521ef9dea84\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:20.440953 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:21:20.440926 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:21:20.517450 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.517426 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj"] Oct 31 19:21:20.520907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.520876 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:20.539076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.538614 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2\" (UID: \"80975e70-dbdf-4a79-a968-c521ef9dea84\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:20.539076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.538657 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2\" (UID: \"80975e70-dbdf-4a79-a968-c521ef9dea84\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:20.539076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.538714 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdm28\" (UniqueName: \"kubernetes.io/projected/80975e70-dbdf-4a79-a968-c521ef9dea84-kube-api-access-kdm28\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2\" (UID: \"80975e70-dbdf-4a79-a968-c521ef9dea84\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:20.539076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.539022 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2\" (UID: \"80975e70-dbdf-4a79-a968-c521ef9dea84\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:20.539459 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.539277 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2\" (UID: \"80975e70-dbdf-4a79-a968-c521ef9dea84\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:20.570024 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.569968 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-b29qw\"" Oct 31 19:21:20.574419 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.574391 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj"] Oct 31 19:21:20.601646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.600961 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdm28\" (UniqueName: \"kubernetes.io/projected/80975e70-dbdf-4a79-a968-c521ef9dea84-kube-api-access-kdm28\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2\" (UID: \"80975e70-dbdf-4a79-a968-c521ef9dea84\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:20.629183 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.629156 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:20.639411 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.639385 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj\" (UID: \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:20.639533 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.639424 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj\" (UID: \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:20.639533 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.639479 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zb7\" (UniqueName: \"kubernetes.io/projected/5799e2a1-3e2d-44b0-8d89-85a7563753d0-kube-api-access-w2zb7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj\" (UID: \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:20.740278 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.740242 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj\" (UID: \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:20.740436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.740294 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj\" (UID: \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:20.740436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.740360 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zb7\" (UniqueName: \"kubernetes.io/projected/5799e2a1-3e2d-44b0-8d89-85a7563753d0-kube-api-access-w2zb7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj\" (UID: \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:20.740717 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.740657 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj\" (UID: \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:20.740996 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.740975 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj\" (UID: \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:20.793812 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.793783 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zb7\" (UniqueName: \"kubernetes.io/projected/5799e2a1-3e2d-44b0-8d89-85a7563753d0-kube-api-access-w2zb7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj\" (UID: \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:20.839949 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:20.839862 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:21.025838 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:21.025108 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2"] Oct 31 19:21:21.029954 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:21:21.029912 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80975e70_dbdf_4a79_a968_c521ef9dea84.slice/crio-2fc4d623388a9c8001d2c497bdc9fc18a3f4788e0373520b13efc0a56891d9a3 WatchSource:0}: Error finding container 2fc4d623388a9c8001d2c497bdc9fc18a3f4788e0373520b13efc0a56891d9a3: Status 404 returned error can't find the container with id 2fc4d623388a9c8001d2c497bdc9fc18a3f4788e0373520b13efc0a56891d9a3 Oct 31 19:21:21.268253 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:21:21.268215 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5799e2a1_3e2d_44b0_8d89_85a7563753d0.slice/crio-227bcf294784836fbebc5b4121b5dad4d3348482180129f94fb7eba2f1c98019 WatchSource:0}: Error finding container 227bcf294784836fbebc5b4121b5dad4d3348482180129f94fb7eba2f1c98019: Status 404 returned error can't find the container with id 227bcf294784836fbebc5b4121b5dad4d3348482180129f94fb7eba2f1c98019 Oct 31 19:21:21.273944 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:21.273871 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj"] Oct 31 19:21:21.410609 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:21.410570 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" event={"ID":"80975e70-dbdf-4a79-a968-c521ef9dea84","Type":"ContainerStarted","Data":"2fc4d623388a9c8001d2c497bdc9fc18a3f4788e0373520b13efc0a56891d9a3"} Oct 31 19:21:21.413094 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:21.413059 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" event={"ID":"5799e2a1-3e2d-44b0-8d89-85a7563753d0","Type":"ContainerStarted","Data":"227bcf294784836fbebc5b4121b5dad4d3348482180129f94fb7eba2f1c98019"} Oct 31 19:21:29.483757 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:29.483717 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" event={"ID":"37f99e10-708a-40c3-bddd-19d13f813cb7","Type":"ContainerStarted","Data":"21b9d3f670b915591316dcb5c858b49d392220b88055c9097327e68d22564321"} Oct 31 19:21:30.490812 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:30.490724 6107 generic.go:356] "Generic (PLEG): container finished" podID="5799e2a1-3e2d-44b0-8d89-85a7563753d0" containerID="dfbba6319dd55e940471fe3906d065da2f1aa438942942183b0407cf0c60b775" exitCode=0 Oct 31 19:21:30.491243 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:30.490831 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" event={"ID":"5799e2a1-3e2d-44b0-8d89-85a7563753d0","Type":"ContainerDied","Data":"dfbba6319dd55e940471fe3906d065da2f1aa438942942183b0407cf0c60b775"} Oct 31 19:21:30.494063 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:30.493854 6107 generic.go:356] "Generic (PLEG): container finished" podID="80975e70-dbdf-4a79-a968-c521ef9dea84" containerID="157bc60bb734b7bf73434e29a62334b72ea2c35b890e757128e973c7971b9589" exitCode=0 Oct 31 19:21:30.494063 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:30.493954 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" event={"ID":"80975e70-dbdf-4a79-a968-c521ef9dea84","Type":"ContainerDied","Data":"157bc60bb734b7bf73434e29a62334b72ea2c35b890e757128e973c7971b9589"} Oct 31 19:21:30.499293 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:30.499204 6107 generic.go:356] "Generic (PLEG): container finished" podID="37f99e10-708a-40c3-bddd-19d13f813cb7" containerID="21b9d3f670b915591316dcb5c858b49d392220b88055c9097327e68d22564321" exitCode=0 Oct 31 19:21:30.499293 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:30.499238 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" event={"ID":"37f99e10-708a-40c3-bddd-19d13f813cb7","Type":"ContainerDied","Data":"21b9d3f670b915591316dcb5c858b49d392220b88055c9097327e68d22564321"} Oct 31 19:21:32.513036 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:32.512898 6107 generic.go:356] "Generic (PLEG): container finished" podID="4d212d46-98dd-4b4c-a898-629d409f548e" containerID="f8869308a237fb3e94ad2fedcb24a49e5976ea0c513c93df02cd755dae3b04e9" exitCode=0 Oct 31 19:21:32.513036 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:32.512978 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" event={"ID":"4d212d46-98dd-4b4c-a898-629d409f548e","Type":"ContainerDied","Data":"f8869308a237fb3e94ad2fedcb24a49e5976ea0c513c93df02cd755dae3b04e9"} Oct 31 19:21:32.515067 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:32.515041 6107 generic.go:356] "Generic (PLEG): container finished" podID="b269313c-091e-4d9c-a229-5c7f1e71ea30" containerID="368e23d652ac2e0ad14b25ab28e8b8a320f7ad72cf3be90fcce71d37327f6fe9" exitCode=0 Oct 31 19:21:32.515322 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:32.515302 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" event={"ID":"b269313c-091e-4d9c-a229-5c7f1e71ea30","Type":"ContainerDied","Data":"368e23d652ac2e0ad14b25ab28e8b8a320f7ad72cf3be90fcce71d37327f6fe9"} Oct 31 19:21:32.518448 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:32.518425 6107 generic.go:356] "Generic (PLEG): container finished" podID="15380b16-b44d-4c07-972f-30afada962ce" containerID="dae17b77c7570c0dc270b4aa7e568e30418b55161810e30ced9701dd9c5f5122" exitCode=0 Oct 31 19:21:32.518602 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:32.518465 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" event={"ID":"15380b16-b44d-4c07-972f-30afada962ce","Type":"ContainerDied","Data":"dae17b77c7570c0dc270b4aa7e568e30418b55161810e30ced9701dd9c5f5122"} Oct 31 19:21:32.521441 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:32.521334 6107 generic.go:356] "Generic (PLEG): container finished" podID="37f99e10-708a-40c3-bddd-19d13f813cb7" containerID="27e95fde78448daca648209f81a353c34ced5f9365c252f89517c2557b967771" exitCode=0 Oct 31 19:21:32.521441 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:32.521358 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" event={"ID":"37f99e10-708a-40c3-bddd-19d13f813cb7","Type":"ContainerDied","Data":"27e95fde78448daca648209f81a353c34ced5f9365c252f89517c2557b967771"} Oct 31 19:21:32.524407 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:32.524333 6107 generic.go:356] "Generic (PLEG): container finished" podID="8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" containerID="ff2b0109bd90ebdda40658d518b8fb699d6fac475b54722fcb2bf69d97693188" exitCode=0 Oct 31 19:21:32.524407 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:32.524374 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" event={"ID":"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7","Type":"ContainerDied","Data":"ff2b0109bd90ebdda40658d518b8fb699d6fac475b54722fcb2bf69d97693188"} Oct 31 19:21:34.536790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:34.536671 6107 generic.go:356] "Generic (PLEG): container finished" podID="8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" containerID="6c9eeb6a1394fff650810dfd67345ecf64d0de9e45685cbf55886d5c56eabca3" exitCode=0 Oct 31 19:21:34.536790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:34.536731 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" event={"ID":"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7","Type":"ContainerDied","Data":"6c9eeb6a1394fff650810dfd67345ecf64d0de9e45685cbf55886d5c56eabca3"} Oct 31 19:21:34.539258 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:34.539186 6107 generic.go:356] "Generic (PLEG): container finished" podID="5799e2a1-3e2d-44b0-8d89-85a7563753d0" containerID="41b182926a3005f1976598001e01d4aae5b74e383099e445ab019c9531bf22d4" exitCode=0 Oct 31 19:21:34.539413 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:34.539287 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" event={"ID":"5799e2a1-3e2d-44b0-8d89-85a7563753d0","Type":"ContainerDied","Data":"41b182926a3005f1976598001e01d4aae5b74e383099e445ab019c9531bf22d4"} Oct 31 19:21:34.543191 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:34.542635 6107 generic.go:356] "Generic (PLEG): container finished" podID="b269313c-091e-4d9c-a229-5c7f1e71ea30" containerID="324f55c5f890ec3c7fc15f081f732862ac4119ca0b7e3668290cbbcd0d240e73" exitCode=0 Oct 31 19:21:34.543191 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:34.542708 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" event={"ID":"b269313c-091e-4d9c-a229-5c7f1e71ea30","Type":"ContainerDied","Data":"324f55c5f890ec3c7fc15f081f732862ac4119ca0b7e3668290cbbcd0d240e73"} Oct 31 19:21:34.545918 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:34.545861 6107 generic.go:356] "Generic (PLEG): container finished" podID="80975e70-dbdf-4a79-a968-c521ef9dea84" containerID="9e622ea6a7c9736d3578cfb6555d48ea201a94010dc307b477eb935934afe0fc" exitCode=0 Oct 31 19:21:34.545990 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:34.545915 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" event={"ID":"80975e70-dbdf-4a79-a968-c521ef9dea84","Type":"ContainerDied","Data":"9e622ea6a7c9736d3578cfb6555d48ea201a94010dc307b477eb935934afe0fc"} Oct 31 19:21:35.553295 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:35.553258 6107 generic.go:356] "Generic (PLEG): container finished" podID="4d212d46-98dd-4b4c-a898-629d409f548e" containerID="62037680b8db54e6dc7fb9e8c7534558fb972c43be6203bdf0fb22149796309c" exitCode=0 Oct 31 19:21:35.553907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:35.553353 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" event={"ID":"4d212d46-98dd-4b4c-a898-629d409f548e","Type":"ContainerDied","Data":"62037680b8db54e6dc7fb9e8c7534558fb972c43be6203bdf0fb22149796309c"} Oct 31 19:21:35.555508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:35.555417 6107 generic.go:356] "Generic (PLEG): container finished" podID="15380b16-b44d-4c07-972f-30afada962ce" containerID="d79c77ecd40ce9fb37aa29f2d394f3fe8de1c33e6e9a34e5082ecefe1522fc30" exitCode=0 Oct 31 19:21:35.555508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:35.555459 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" event={"ID":"15380b16-b44d-4c07-972f-30afada962ce","Type":"ContainerDied","Data":"d79c77ecd40ce9fb37aa29f2d394f3fe8de1c33e6e9a34e5082ecefe1522fc30"} Oct 31 19:21:47.647342 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:47.647255 6107 generic.go:356] "Generic (PLEG): container finished" podID="37f99e10-708a-40c3-bddd-19d13f813cb7" containerID="d32866823fdf4fd53b1c2601ddaaa7ac446abbfef3d1424d28f34e07c68b3a19" exitCode=0 Oct 31 19:21:47.647933 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:47.647367 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" event={"ID":"37f99e10-708a-40c3-bddd-19d13f813cb7","Type":"ContainerDied","Data":"d32866823fdf4fd53b1c2601ddaaa7ac446abbfef3d1424d28f34e07c68b3a19"} Oct 31 19:21:48.825254 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:48.825223 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:48.937693 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:48.936512 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-util\" (UniqueName: \"kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-util\") pod \"37f99e10-708a-40c3-bddd-19d13f813cb7\" (UID: \"37f99e10-708a-40c3-bddd-19d13f813cb7\") " Oct 31 19:21:48.937693 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:48.936584 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f99e10-708a-40c3-bddd-19d13f813cb7-kube-api-access-tdnjv\" (UniqueName: \"kubernetes.io/projected/37f99e10-708a-40c3-bddd-19d13f813cb7-kube-api-access-tdnjv\") pod \"37f99e10-708a-40c3-bddd-19d13f813cb7\" (UID: \"37f99e10-708a-40c3-bddd-19d13f813cb7\") " Oct 31 19:21:48.937693 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:48.936699 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-bundle\") pod \"37f99e10-708a-40c3-bddd-19d13f813cb7\" (UID: \"37f99e10-708a-40c3-bddd-19d13f813cb7\") " Oct 31 19:21:48.941325 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:48.941263 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-bundle" pod "37f99e10-708a-40c3-bddd-19d13f813cb7" (UID: "37f99e10-708a-40c3-bddd-19d13f813cb7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:48.943109 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:48.943070 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f99e10-708a-40c3-bddd-19d13f813cb7-kube-api-access-tdnjv" pod "37f99e10-708a-40c3-bddd-19d13f813cb7" (UID: "37f99e10-708a-40c3-bddd-19d13f813cb7"). InnerVolumeSpecName "kube-api-access-tdnjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:21:48.953525 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:48.953476 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-util" pod "37f99e10-708a-40c3-bddd-19d13f813cb7" (UID: "37f99e10-708a-40c3-bddd-19d13f813cb7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:49.038299 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:49.038247 6107 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-bundle\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:49.038469 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:49.038284 6107 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f99e10-708a-40c3-bddd-19d13f813cb7-util\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:49.038469 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:49.038454 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdnjv\" (UniqueName: \"kubernetes.io/projected/37f99e10-708a-40c3-bddd-19d13f813cb7-kube-api-access-tdnjv\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:49.667520 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:49.667482 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" event={"ID":"5799e2a1-3e2d-44b0-8d89-85a7563753d0","Type":"ContainerStarted","Data":"f9c31dcf9000065d4b8a278e1e2927dc20120dd3ba1704479db16cf3f785d849"} Oct 31 19:21:49.671759 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:49.671732 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" event={"ID":"b269313c-091e-4d9c-a229-5c7f1e71ea30","Type":"ContainerStarted","Data":"f4c122de731b254ff5b06da65cfb3008692656e9291d07dad218778a36736c5a"} Oct 31 19:21:49.674547 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:49.674525 6107 generic.go:356] "Generic (PLEG): container finished" podID="80975e70-dbdf-4a79-a968-c521ef9dea84" containerID="aa163162feed8cb95750735c6da92af9f2dc978f9ce1ea4b3f7e8d76a1a1a12c" exitCode=0 Oct 31 19:21:49.674655 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:49.674597 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" event={"ID":"80975e70-dbdf-4a79-a968-c521ef9dea84","Type":"ContainerDied","Data":"aa163162feed8cb95750735c6da92af9f2dc978f9ce1ea4b3f7e8d76a1a1a12c"} Oct 31 19:21:49.676800 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:49.676759 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" event={"ID":"37f99e10-708a-40c3-bddd-19d13f813cb7","Type":"ContainerDied","Data":"059fdda7b73a98931908357829b065944eca350a931c2b1e685869062f30e4f1"} Oct 31 19:21:49.676979 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:49.676848 6107 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="059fdda7b73a98931908357829b065944eca350a931c2b1e685869062f30e4f1" Oct 31 19:21:49.676979 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:49.676977 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tdxtc" Oct 31 19:21:49.777740 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:49.777675 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" podStartSLOduration=2.2937986329999998 podStartE2EDuration="29.777637178s" totalImagesPullingTime="27.483838545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:21:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:21:49.77744405 +0000 UTC m=+281.410837973" watchObservedRunningTime="2026-10-31 19:21:49.777637178 +0000 UTC m=+281.411031086" Oct 31 19:21:49.886852 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:49.886743 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" podStartSLOduration=2.503112312 podStartE2EDuration="31.886711558s" totalImagesPullingTime="29.383599246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:21:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:21:49.885939686 +0000 UTC m=+281.519333596" watchObservedRunningTime="2026-10-31 19:21:49.886711558 +0000 UTC m=+281.520105467" Oct 31 19:21:50.401133 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:21:50.401059 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:21:50.682441 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.682415 6107 generic.go:356] "Generic (PLEG): container finished" podID="4d212d46-98dd-4b4c-a898-629d409f548e" containerID="83ebfccf554ce0aaf2b32d938887eb7795dd23290bf9c217f5b8cb3e4cbae7cc" exitCode=0 Oct 31 19:21:50.682557 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.682497 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" event={"ID":"4d212d46-98dd-4b4c-a898-629d409f548e","Type":"ContainerDied","Data":"83ebfccf554ce0aaf2b32d938887eb7795dd23290bf9c217f5b8cb3e4cbae7cc"} Oct 31 19:21:50.684477 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.684454 6107 generic.go:356] "Generic (PLEG): container finished" podID="5799e2a1-3e2d-44b0-8d89-85a7563753d0" containerID="f9c31dcf9000065d4b8a278e1e2927dc20120dd3ba1704479db16cf3f785d849" exitCode=0 Oct 31 19:21:50.684588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.684518 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" event={"ID":"5799e2a1-3e2d-44b0-8d89-85a7563753d0","Type":"ContainerDied","Data":"f9c31dcf9000065d4b8a278e1e2927dc20120dd3ba1704479db16cf3f785d849"} Oct 31 19:21:50.686454 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.686431 6107 generic.go:356] "Generic (PLEG): container finished" podID="b269313c-091e-4d9c-a229-5c7f1e71ea30" containerID="f4c122de731b254ff5b06da65cfb3008692656e9291d07dad218778a36736c5a" exitCode=0 Oct 31 19:21:50.686538 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.686513 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" event={"ID":"b269313c-091e-4d9c-a229-5c7f1e71ea30","Type":"ContainerDied","Data":"f4c122de731b254ff5b06da65cfb3008692656e9291d07dad218778a36736c5a"} Oct 31 19:21:50.688367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.688345 6107 generic.go:356] "Generic (PLEG): container finished" podID="15380b16-b44d-4c07-972f-30afada962ce" containerID="5f601f4481ef546ae17c5ce6abbe3ccd5f55873f7d8a82f3f405474725b05ae8" exitCode=0 Oct 31 19:21:50.688468 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.688427 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" event={"ID":"15380b16-b44d-4c07-972f-30afada962ce","Type":"ContainerDied","Data":"5f601f4481ef546ae17c5ce6abbe3ccd5f55873f7d8a82f3f405474725b05ae8"} Oct 31 19:21:50.690200 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.690180 6107 generic.go:356] "Generic (PLEG): container finished" podID="8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" containerID="3614aab7740838303ccb14bddca280073b11338161c4808375a2d05d6e278e56" exitCode=0 Oct 31 19:21:50.690282 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.690257 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" event={"ID":"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7","Type":"ContainerDied","Data":"3614aab7740838303ccb14bddca280073b11338161c4808375a2d05d6e278e56"} Oct 31 19:21:50.806105 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.806085 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:50.853078 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.853058 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-bundle\" (UniqueName: \"kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-bundle\") pod \"80975e70-dbdf-4a79-a968-c521ef9dea84\" (UID: \"80975e70-dbdf-4a79-a968-c521ef9dea84\") " Oct 31 19:21:50.853188 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.853085 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80975e70-dbdf-4a79-a968-c521ef9dea84-kube-api-access-kdm28\" (UniqueName: \"kubernetes.io/projected/80975e70-dbdf-4a79-a968-c521ef9dea84-kube-api-access-kdm28\") pod \"80975e70-dbdf-4a79-a968-c521ef9dea84\" (UID: \"80975e70-dbdf-4a79-a968-c521ef9dea84\") " Oct 31 19:21:50.853188 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.853106 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-util\" (UniqueName: \"kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-util\") pod \"80975e70-dbdf-4a79-a968-c521ef9dea84\" (UID: \"80975e70-dbdf-4a79-a968-c521ef9dea84\") " Oct 31 19:21:50.853464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.853439 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-bundle" pod "80975e70-dbdf-4a79-a968-c521ef9dea84" (UID: "80975e70-dbdf-4a79-a968-c521ef9dea84"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:50.855049 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.855026 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80975e70-dbdf-4a79-a968-c521ef9dea84-kube-api-access-kdm28" pod "80975e70-dbdf-4a79-a968-c521ef9dea84" (UID: "80975e70-dbdf-4a79-a968-c521ef9dea84"). InnerVolumeSpecName "kube-api-access-kdm28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:21:50.857913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.857877 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-util" pod "80975e70-dbdf-4a79-a968-c521ef9dea84" (UID: "80975e70-dbdf-4a79-a968-c521ef9dea84"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:50.953839 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.953772 6107 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-bundle\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:50.953839 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.953793 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdm28\" (UniqueName: \"kubernetes.io/projected/80975e70-dbdf-4a79-a968-c521ef9dea84-kube-api-access-kdm28\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:50.953839 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:50.953802 6107 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80975e70-dbdf-4a79-a968-c521ef9dea84-util\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:51.695840 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.695813 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" Oct 31 19:21:51.696022 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.695865 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644snhq2" event={"ID":"80975e70-dbdf-4a79-a968-c521ef9dea84","Type":"ContainerDied","Data":"2fc4d623388a9c8001d2c497bdc9fc18a3f4788e0373520b13efc0a56891d9a3"} Oct 31 19:21:51.696022 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.695919 6107 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fc4d623388a9c8001d2c497bdc9fc18a3f4788e0373520b13efc0a56891d9a3" Oct 31 19:21:51.895205 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.895179 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:51.897431 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.897413 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:51.900787 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.900768 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:51.904271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.904254 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:51.906926 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.906910 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:51.962988 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.962914 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-bundle\") pod \"b269313c-091e-4d9c-a229-5c7f1e71ea30\" (UID: \"b269313c-091e-4d9c-a229-5c7f1e71ea30\") " Oct 31 19:21:51.962988 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.962947 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-util\" (UniqueName: \"kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-util\") pod \"4d212d46-98dd-4b4c-a898-629d409f548e\" (UID: \"4d212d46-98dd-4b4c-a898-629d409f548e\") " Oct 31 19:21:51.962988 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.962972 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b269313c-091e-4d9c-a229-5c7f1e71ea30-kube-api-access-wqvwj\" (UniqueName: \"kubernetes.io/projected/b269313c-091e-4d9c-a229-5c7f1e71ea30-kube-api-access-wqvwj\") pod \"b269313c-091e-4d9c-a229-5c7f1e71ea30\" (UID: \"b269313c-091e-4d9c-a229-5c7f1e71ea30\") " Oct 31 19:21:51.963531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963008 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-bundle\") pod \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\" (UID: \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\") " Oct 31 19:21:51.963531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963035 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-bundle\" (UniqueName: \"kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-bundle\") pod \"15380b16-b44d-4c07-972f-30afada962ce\" (UID: \"15380b16-b44d-4c07-972f-30afada962ce\") " Oct 31 19:21:51.963531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963072 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d212d46-98dd-4b4c-a898-629d409f548e-kube-api-access-jr467\" (UniqueName: \"kubernetes.io/projected/4d212d46-98dd-4b4c-a898-629d409f548e-kube-api-access-jr467\") pod \"4d212d46-98dd-4b4c-a898-629d409f548e\" (UID: \"4d212d46-98dd-4b4c-a898-629d409f548e\") " Oct 31 19:21:51.963531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963101 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15380b16-b44d-4c07-972f-30afada962ce-kube-api-access-bmkr7\" (UniqueName: \"kubernetes.io/projected/15380b16-b44d-4c07-972f-30afada962ce-kube-api-access-bmkr7\") pod \"15380b16-b44d-4c07-972f-30afada962ce\" (UID: \"15380b16-b44d-4c07-972f-30afada962ce\") " Oct 31 19:21:51.963531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963136 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-kube-api-access-jxmg9\" (UniqueName: \"kubernetes.io/projected/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-kube-api-access-jxmg9\") pod \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\" (UID: \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\") " Oct 31 19:21:51.963531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963163 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-bundle\") pod \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\" (UID: \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\") " Oct 31 19:21:51.963531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963195 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-util\" (UniqueName: \"kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-util\") pod \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\" (UID: \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\") " Oct 31 19:21:51.963531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963232 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-bundle\") pod \"4d212d46-98dd-4b4c-a898-629d409f548e\" (UID: \"4d212d46-98dd-4b4c-a898-629d409f548e\") " Oct 31 19:21:51.963531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963258 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-util\" (UniqueName: \"kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-util\") pod \"b269313c-091e-4d9c-a229-5c7f1e71ea30\" (UID: \"b269313c-091e-4d9c-a229-5c7f1e71ea30\") " Oct 31 19:21:51.963531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963291 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-util\" (UniqueName: \"kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-util\") pod \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\" (UID: \"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7\") " Oct 31 19:21:51.963531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963338 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5799e2a1-3e2d-44b0-8d89-85a7563753d0-kube-api-access-w2zb7\" (UniqueName: \"kubernetes.io/projected/5799e2a1-3e2d-44b0-8d89-85a7563753d0-kube-api-access-w2zb7\") pod \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\" (UID: \"5799e2a1-3e2d-44b0-8d89-85a7563753d0\") " Oct 31 19:21:51.963531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963386 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-util\" (UniqueName: \"kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-util\") pod \"15380b16-b44d-4c07-972f-30afada962ce\" (UID: \"15380b16-b44d-4c07-972f-30afada962ce\") " Oct 31 19:21:51.964138 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.963675 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-bundle" pod "15380b16-b44d-4c07-972f-30afada962ce" (UID: "15380b16-b44d-4c07-972f-30afada962ce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:51.964292 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.964260 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-bundle" pod "8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" (UID: "8d9eb5b7-1463-4dba-9b7a-f4f73052abc7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:51.964425 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.964364 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-bundle" pod "4d212d46-98dd-4b4c-a898-629d409f548e" (UID: "4d212d46-98dd-4b4c-a898-629d409f548e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:51.964542 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.964520 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-bundle" pod "b269313c-091e-4d9c-a229-5c7f1e71ea30" (UID: "b269313c-091e-4d9c-a229-5c7f1e71ea30"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:51.965739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.965716 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-bundle" pod "5799e2a1-3e2d-44b0-8d89-85a7563753d0" (UID: "5799e2a1-3e2d-44b0-8d89-85a7563753d0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:51.966934 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.966907 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-kube-api-access-jxmg9" pod "8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" (UID: "8d9eb5b7-1463-4dba-9b7a-f4f73052abc7"). InnerVolumeSpecName "kube-api-access-jxmg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:21:51.967394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.967369 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5799e2a1-3e2d-44b0-8d89-85a7563753d0-kube-api-access-w2zb7" pod "5799e2a1-3e2d-44b0-8d89-85a7563753d0" (UID: "5799e2a1-3e2d-44b0-8d89-85a7563753d0"). InnerVolumeSpecName "kube-api-access-w2zb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:21:51.967703 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.967682 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b269313c-091e-4d9c-a229-5c7f1e71ea30-kube-api-access-wqvwj" pod "b269313c-091e-4d9c-a229-5c7f1e71ea30" (UID: "b269313c-091e-4d9c-a229-5c7f1e71ea30"). InnerVolumeSpecName "kube-api-access-wqvwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:21:51.967703 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.967679 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d212d46-98dd-4b4c-a898-629d409f548e-kube-api-access-jr467" pod "4d212d46-98dd-4b4c-a898-629d409f548e" (UID: "4d212d46-98dd-4b4c-a898-629d409f548e"). InnerVolumeSpecName "kube-api-access-jr467". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:21:51.969190 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.969160 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15380b16-b44d-4c07-972f-30afada962ce-kube-api-access-bmkr7" pod "15380b16-b44d-4c07-972f-30afada962ce" (UID: "15380b16-b44d-4c07-972f-30afada962ce"). InnerVolumeSpecName "kube-api-access-bmkr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:21:51.972463 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.972443 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-util" pod "8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" (UID: "8d9eb5b7-1463-4dba-9b7a-f4f73052abc7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:51.972563 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.972502 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-util" pod "15380b16-b44d-4c07-972f-30afada962ce" (UID: "15380b16-b44d-4c07-972f-30afada962ce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:51.972769 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.972748 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-util" pod "b269313c-091e-4d9c-a229-5c7f1e71ea30" (UID: "b269313c-091e-4d9c-a229-5c7f1e71ea30"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:51.972962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.972945 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-util" pod "4d212d46-98dd-4b4c-a898-629d409f548e" (UID: "4d212d46-98dd-4b4c-a898-629d409f548e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:51.973460 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:51.973439 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-util" pod "5799e2a1-3e2d-44b0-8d89-85a7563753d0" (UID: "5799e2a1-3e2d-44b0-8d89-85a7563753d0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:52.064150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064124 6107 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-bundle\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064147 6107 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5799e2a1-3e2d-44b0-8d89-85a7563753d0-util\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064156 6107 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-bundle\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064164 6107 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-util\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064171 6107 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-util\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064180 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2zb7\" (UniqueName: \"kubernetes.io/projected/5799e2a1-3e2d-44b0-8d89-85a7563753d0-kube-api-access-w2zb7\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064189 6107 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-util\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064197 6107 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b269313c-091e-4d9c-a229-5c7f1e71ea30-bundle\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064206 6107 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d212d46-98dd-4b4c-a898-629d409f548e-util\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064214 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqvwj\" (UniqueName: \"kubernetes.io/projected/b269313c-091e-4d9c-a229-5c7f1e71ea30-kube-api-access-wqvwj\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064222 6107 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-bundle\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064230 6107 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15380b16-b44d-4c07-972f-30afada962ce-bundle\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064237 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr467\" (UniqueName: \"kubernetes.io/projected/4d212d46-98dd-4b4c-a898-629d409f548e-kube-api-access-jr467\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064246 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmkr7\" (UniqueName: \"kubernetes.io/projected/15380b16-b44d-4c07-972f-30afada962ce-kube-api-access-bmkr7\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.064366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.064254 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxmg9\" (UniqueName: \"kubernetes.io/projected/8d9eb5b7-1463-4dba-9b7a-f4f73052abc7-kube-api-access-jxmg9\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:52.701811 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.701785 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" Oct 31 19:21:52.702006 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.701774 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah9z45" event={"ID":"b269313c-091e-4d9c-a229-5c7f1e71ea30","Type":"ContainerDied","Data":"80878711278e1a63c0d06912c306b30b3dc8c688deb206dda025b8b69c3f06c7"} Oct 31 19:21:52.702006 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.701962 6107 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80878711278e1a63c0d06912c306b30b3dc8c688deb206dda025b8b69c3f06c7" Oct 31 19:21:52.704256 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.703793 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" Oct 31 19:21:52.704256 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.703815 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66kpxg" event={"ID":"15380b16-b44d-4c07-972f-30afada962ce","Type":"ContainerDied","Data":"2faebe344d4200f9a211183c10130f7bd39a3e26dc76df0fc20f927856256197"} Oct 31 19:21:52.704256 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.703857 6107 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2faebe344d4200f9a211183c10130f7bd39a3e26dc76df0fc20f927856256197" Oct 31 19:21:52.705914 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.705875 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" Oct 31 19:21:52.706027 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.705908 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76d45v" event={"ID":"8d9eb5b7-1463-4dba-9b7a-f4f73052abc7","Type":"ContainerDied","Data":"d1685e7e229a4f64e7fdd8cc210b0ba96baffef224f7a80fcca2f8a42774ea04"} Oct 31 19:21:52.706027 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.705951 6107 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1685e7e229a4f64e7fdd8cc210b0ba96baffef224f7a80fcca2f8a42774ea04" Oct 31 19:21:52.707831 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.707793 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" event={"ID":"4d212d46-98dd-4b4c-a898-629d409f548e","Type":"ContainerDied","Data":"4458e928a7201a7f60b257444febf9eafd9be93ccc4fe3de5b9ebd2195d34085"} Oct 31 19:21:52.707831 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.707821 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd5xrt" Oct 31 19:21:52.708018 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.707833 6107 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4458e928a7201a7f60b257444febf9eafd9be93ccc4fe3de5b9ebd2195d34085" Oct 31 19:21:52.709830 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.709808 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" event={"ID":"5799e2a1-3e2d-44b0-8d89-85a7563753d0","Type":"ContainerDied","Data":"227bcf294784836fbebc5b4121b5dad4d3348482180129f94fb7eba2f1c98019"} Oct 31 19:21:52.709954 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.709841 6107 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="227bcf294784836fbebc5b4121b5dad4d3348482180129f94fb7eba2f1c98019" Oct 31 19:21:52.710015 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:52.709989 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c65ntj" Oct 31 19:21:55.389119 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389089 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc"] Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389477 6107 cpu_manager.go:414] "containerMap: removing container" podUID="b269313c-091e-4d9c-a229-5c7f1e71ea30" containerName="pull" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389493 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b269313c-091e-4d9c-a229-5c7f1e71ea30" containerName="pull" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389509 6107 cpu_manager.go:414] "containerMap: removing container" podUID="8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" containerName="pull" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389516 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" containerName="pull" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389526 6107 cpu_manager.go:414] "containerMap: removing container" podUID="b269313c-091e-4d9c-a229-5c7f1e71ea30" containerName="extract" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389533 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b269313c-091e-4d9c-a229-5c7f1e71ea30" containerName="extract" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389543 6107 cpu_manager.go:414] "containerMap: removing container" podUID="4d212d46-98dd-4b4c-a898-629d409f548e" containerName="extract" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389549 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d212d46-98dd-4b4c-a898-629d409f548e" containerName="extract" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389559 6107 cpu_manager.go:414] "containerMap: removing container" podUID="80975e70-dbdf-4a79-a968-c521ef9dea84" containerName="pull" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389565 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80975e70-dbdf-4a79-a968-c521ef9dea84" containerName="pull" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389579 6107 cpu_manager.go:414] "containerMap: removing container" podUID="5799e2a1-3e2d-44b0-8d89-85a7563753d0" containerName="extract" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389585 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5799e2a1-3e2d-44b0-8d89-85a7563753d0" containerName="extract" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389603 6107 cpu_manager.go:414] "containerMap: removing container" podUID="8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" containerName="util" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389610 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" containerName="util" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389622 6107 cpu_manager.go:414] "containerMap: removing container" podUID="15380b16-b44d-4c07-972f-30afada962ce" containerName="extract" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389629 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15380b16-b44d-4c07-972f-30afada962ce" containerName="extract" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389639 6107 cpu_manager.go:414] "containerMap: removing container" podUID="37f99e10-708a-40c3-bddd-19d13f813cb7" containerName="util" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389645 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f99e10-708a-40c3-bddd-19d13f813cb7" containerName="util" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389655 6107 cpu_manager.go:414] "containerMap: removing container" podUID="15380b16-b44d-4c07-972f-30afada962ce" containerName="util" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389661 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15380b16-b44d-4c07-972f-30afada962ce" containerName="util" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389673 6107 cpu_manager.go:414] "containerMap: removing container" podUID="4d212d46-98dd-4b4c-a898-629d409f548e" containerName="util" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389679 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d212d46-98dd-4b4c-a898-629d409f548e" containerName="util" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389687 6107 cpu_manager.go:414] "containerMap: removing container" podUID="4d212d46-98dd-4b4c-a898-629d409f548e" containerName="pull" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389692 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d212d46-98dd-4b4c-a898-629d409f548e" containerName="pull" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389707 6107 cpu_manager.go:414] "containerMap: removing container" podUID="37f99e10-708a-40c3-bddd-19d13f813cb7" containerName="pull" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389716 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f99e10-708a-40c3-bddd-19d13f813cb7" containerName="pull" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389725 6107 cpu_manager.go:414] "containerMap: removing container" podUID="37f99e10-708a-40c3-bddd-19d13f813cb7" containerName="extract" Oct 31 19:21:55.389712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389732 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f99e10-708a-40c3-bddd-19d13f813cb7" containerName="extract" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389741 6107 cpu_manager.go:414] "containerMap: removing container" podUID="5799e2a1-3e2d-44b0-8d89-85a7563753d0" containerName="pull" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389749 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5799e2a1-3e2d-44b0-8d89-85a7563753d0" containerName="pull" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389761 6107 cpu_manager.go:414] "containerMap: removing container" podUID="8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" containerName="extract" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389767 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" containerName="extract" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389780 6107 cpu_manager.go:414] "containerMap: removing container" podUID="15380b16-b44d-4c07-972f-30afada962ce" containerName="pull" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389786 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15380b16-b44d-4c07-972f-30afada962ce" containerName="pull" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389795 6107 cpu_manager.go:414] "containerMap: removing container" podUID="80975e70-dbdf-4a79-a968-c521ef9dea84" containerName="extract" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389801 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80975e70-dbdf-4a79-a968-c521ef9dea84" containerName="extract" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389814 6107 cpu_manager.go:414] "containerMap: removing container" podUID="80975e70-dbdf-4a79-a968-c521ef9dea84" containerName="util" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389821 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80975e70-dbdf-4a79-a968-c521ef9dea84" containerName="util" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389830 6107 cpu_manager.go:414] "containerMap: removing container" podUID="5799e2a1-3e2d-44b0-8d89-85a7563753d0" containerName="util" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389836 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5799e2a1-3e2d-44b0-8d89-85a7563753d0" containerName="util" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389851 6107 cpu_manager.go:414] "containerMap: removing container" podUID="b269313c-091e-4d9c-a229-5c7f1e71ea30" containerName="util" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389858 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b269313c-091e-4d9c-a229-5c7f1e71ea30" containerName="util" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389956 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f99e10-708a-40c3-bddd-19d13f813cb7" containerName="extract" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389970 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d9eb5b7-1463-4dba-9b7a-f4f73052abc7" containerName="extract" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389980 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="80975e70-dbdf-4a79-a968-c521ef9dea84" containerName="extract" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389989 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="5799e2a1-3e2d-44b0-8d89-85a7563753d0" containerName="extract" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.389998 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d212d46-98dd-4b4c-a898-629d409f548e" containerName="extract" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.390008 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="b269313c-091e-4d9c-a229-5c7f1e71ea30" containerName="extract" Oct 31 19:21:55.391146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.390017 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="15380b16-b44d-4c07-972f-30afada962ce" containerName="extract" Oct 31 19:21:55.418069 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.418048 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.468320 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.468293 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 19:21:55.468320 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.468293 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 19:21:55.468320 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.468293 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 19:21:55.468566 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.468333 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 19:21:55.468566 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.468447 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 19:21:55.468566 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.468527 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-7fffj\"" Oct 31 19:21:55.488922 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.488902 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ca954e2-9fde-406a-b805-7d18bec7d577-tmp\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.489020 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.488952 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ca954e2-9fde-406a-b805-7d18bec7d577-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.489020 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.489015 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh5l5\" (UniqueName: \"kubernetes.io/projected/4ca954e2-9fde-406a-b805-7d18bec7d577-kube-api-access-vh5l5\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.489089 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.489074 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ca954e2-9fde-406a-b805-7d18bec7d577-metrics-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.489121 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.489100 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ca954e2-9fde-406a-b805-7d18bec7d577-manifests\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.489164 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.489127 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ca954e2-9fde-406a-b805-7d18bec7d577-webhook-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.515847 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.515822 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc"] Oct 31 19:21:55.589488 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.589454 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ca954e2-9fde-406a-b805-7d18bec7d577-webhook-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.589681 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.589493 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ca954e2-9fde-406a-b805-7d18bec7d577-tmp\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.589681 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.589556 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ca954e2-9fde-406a-b805-7d18bec7d577-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.589681 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.589595 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5l5\" (UniqueName: \"kubernetes.io/projected/4ca954e2-9fde-406a-b805-7d18bec7d577-kube-api-access-vh5l5\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.589681 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.589637 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ca954e2-9fde-406a-b805-7d18bec7d577-metrics-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.589913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.589683 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ca954e2-9fde-406a-b805-7d18bec7d577-manifests\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.590712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.590435 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ca954e2-9fde-406a-b805-7d18bec7d577-manifests\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.592506 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.592477 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ca954e2-9fde-406a-b805-7d18bec7d577-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.592598 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.592548 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ca954e2-9fde-406a-b805-7d18bec7d577-tmp\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.592598 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.592570 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ca954e2-9fde-406a-b805-7d18bec7d577-webhook-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.592709 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.592692 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ca954e2-9fde-406a-b805-7d18bec7d577-metrics-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.661500 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.661433 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh5l5\" (UniqueName: \"kubernetes.io/projected/4ca954e2-9fde-406a-b805-7d18bec7d577-kube-api-access-vh5l5\") pod \"opendatahub-operator-controller-manager-dc8db5648-rvrzc\" (UID: \"4ca954e2-9fde-406a-b805-7d18bec7d577\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:55.767414 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:55.767386 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:21:56.276872 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:56.276837 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc"] Oct 31 19:21:56.325407 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:21:56.325369 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca954e2_9fde_406a_b805_7d18bec7d577.slice/crio-e1bc18fb2427ec1ce40fc29cfb7002f33225a7517aa4bc4676f92d51d4db0314 WatchSource:0}: Error finding container e1bc18fb2427ec1ce40fc29cfb7002f33225a7517aa4bc4676f92d51d4db0314: Status 404 returned error can't find the container with id e1bc18fb2427ec1ce40fc29cfb7002f33225a7517aa4bc4676f92d51d4db0314 Oct 31 19:21:56.730061 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:56.730029 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" event={"ID":"4ca954e2-9fde-406a-b805-7d18bec7d577","Type":"ContainerStarted","Data":"e1bc18fb2427ec1ce40fc29cfb7002f33225a7517aa4bc4676f92d51d4db0314"} Oct 31 19:21:59.745667 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:21:59.745632 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" event={"ID":"4ca954e2-9fde-406a-b805-7d18bec7d577","Type":"ContainerStarted","Data":"d577a18bc05e80ec278eea7e5cdde5a62d4a5ae9d01b4527b49d2658a3a9f224"} Oct 31 19:22:00.071740 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.071706 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp"] Oct 31 19:22:00.076352 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.076335 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp" Oct 31 19:22:00.125919 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.125868 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5g22\" (UniqueName: \"kubernetes.io/projected/6492da2f-bd6a-463a-845b-ce64ec04e20a-kube-api-access-t5g22\") pod \"jobset-operator-674cd59dc5-mfbnp\" (UID: \"6492da2f-bd6a-463a-845b-ce64ec04e20a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp" Oct 31 19:22:00.126051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.125987 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6492da2f-bd6a-463a-845b-ce64ec04e20a-tmp\") pod \"jobset-operator-674cd59dc5-mfbnp\" (UID: \"6492da2f-bd6a-463a-845b-ce64ec04e20a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp" Oct 31 19:22:00.129577 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.129559 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 19:22:00.129678 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.129644 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:22:00.130662 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.130111 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-fpwv4\"" Oct 31 19:22:00.131564 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.131543 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp"] Oct 31 19:22:00.227166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.227131 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5g22\" (UniqueName: \"kubernetes.io/projected/6492da2f-bd6a-463a-845b-ce64ec04e20a-kube-api-access-t5g22\") pod \"jobset-operator-674cd59dc5-mfbnp\" (UID: \"6492da2f-bd6a-463a-845b-ce64ec04e20a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp" Oct 31 19:22:00.227358 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.227179 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6492da2f-bd6a-463a-845b-ce64ec04e20a-tmp\") pod \"jobset-operator-674cd59dc5-mfbnp\" (UID: \"6492da2f-bd6a-463a-845b-ce64ec04e20a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp" Oct 31 19:22:00.227576 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.227556 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6492da2f-bd6a-463a-845b-ce64ec04e20a-tmp\") pod \"jobset-operator-674cd59dc5-mfbnp\" (UID: \"6492da2f-bd6a-463a-845b-ce64ec04e20a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp" Oct 31 19:22:00.281871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.281833 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5g22\" (UniqueName: \"kubernetes.io/projected/6492da2f-bd6a-463a-845b-ce64ec04e20a-kube-api-access-t5g22\") pod \"jobset-operator-674cd59dc5-mfbnp\" (UID: \"6492da2f-bd6a-463a-845b-ce64ec04e20a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp" Oct 31 19:22:00.396765 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.396684 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp" Oct 31 19:22:00.752267 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.752235 6107 generic.go:356] "Generic (PLEG): container finished" podID="4ca954e2-9fde-406a-b805-7d18bec7d577" containerID="d577a18bc05e80ec278eea7e5cdde5a62d4a5ae9d01b4527b49d2658a3a9f224" exitCode=0 Oct 31 19:22:00.752662 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.752313 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" event={"ID":"4ca954e2-9fde-406a-b805-7d18bec7d577","Type":"ContainerDied","Data":"d577a18bc05e80ec278eea7e5cdde5a62d4a5ae9d01b4527b49d2658a3a9f224"} Oct 31 19:22:00.756304 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:00.756278 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp"] Oct 31 19:22:00.759618 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:22:00.759593 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6492da2f_bd6a_463a_845b_ce64ec04e20a.slice/crio-31dd2aef8eecba531dc0615801b5a44f70bf8074891d5fa0821988934c024ef1 WatchSource:0}: Error finding container 31dd2aef8eecba531dc0615801b5a44f70bf8074891d5fa0821988934c024ef1: Status 404 returned error can't find the container with id 31dd2aef8eecba531dc0615801b5a44f70bf8074891d5fa0821988934c024ef1 Oct 31 19:22:01.760794 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:01.760762 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" event={"ID":"4ca954e2-9fde-406a-b805-7d18bec7d577","Type":"ContainerStarted","Data":"aa4a46d7459b3c22d03c7dbf0bacdc3851c6c4f7e499d3e8f4dddf593edbe4e7"} Oct 31 19:22:01.761277 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:01.760900 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:22:01.762032 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:01.762010 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp" event={"ID":"6492da2f-bd6a-463a-845b-ce64ec04e20a","Type":"ContainerStarted","Data":"31dd2aef8eecba531dc0615801b5a44f70bf8074891d5fa0821988934c024ef1"} Oct 31 19:22:01.951988 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:01.951872 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" podStartSLOduration=2.368536469 podStartE2EDuration="6.951857875s" totalImagesPullingTime="4.189686209s" totalInitContainerRuntime="393.635197ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:21:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:01.949019949 +0000 UTC m=+293.582413857" watchObservedRunningTime="2026-10-31 19:22:01.951857875 +0000 UTC m=+293.585251782" Oct 31 19:22:04.784671 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:04.784634 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp" event={"ID":"6492da2f-bd6a-463a-845b-ce64ec04e20a","Type":"ContainerStarted","Data":"2195d9b672b0e59bf44a9f75be8ffcec8f74520ba21923427ef4b43e1977c473"} Oct 31 19:22:04.892708 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:04.892653 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mfbnp" podStartSLOduration=1.892336642 podStartE2EDuration="4.892636936s" totalImagesPullingTime="3.000300294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:04.891556951 +0000 UTC m=+296.524950859" watchObservedRunningTime="2026-10-31 19:22:04.892636936 +0000 UTC m=+296.526030842" Oct 31 19:22:12.770901 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:12.770850 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-rvrzc" Oct 31 19:22:12.828806 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:12.828776 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5"] Oct 31 19:22:12.833288 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:12.833268 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5" Oct 31 19:22:12.884441 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:12.884405 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5"] Oct 31 19:22:12.888076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:12.888054 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:22:12.919712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:12.919690 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 19:22:12.919837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:12.919781 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-dhkxg\"" Oct 31 19:22:12.931084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:12.931060 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zcmr\" (UniqueName: \"kubernetes.io/projected/ae881581-0a50-4775-81e3-d3ee113c7ec2-kube-api-access-4zcmr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5\" (UID: \"ae881581-0a50-4775-81e3-d3ee113c7ec2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5" Oct 31 19:22:12.931142 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:12.931132 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae881581-0a50-4775-81e3-d3ee113c7ec2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5\" (UID: \"ae881581-0a50-4775-81e3-d3ee113c7ec2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5" Oct 31 19:22:13.032367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:13.032293 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zcmr\" (UniqueName: \"kubernetes.io/projected/ae881581-0a50-4775-81e3-d3ee113c7ec2-kube-api-access-4zcmr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5\" (UID: \"ae881581-0a50-4775-81e3-d3ee113c7ec2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5" Oct 31 19:22:13.032367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:13.032354 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae881581-0a50-4775-81e3-d3ee113c7ec2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5\" (UID: \"ae881581-0a50-4775-81e3-d3ee113c7ec2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5" Oct 31 19:22:13.032656 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:13.032640 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae881581-0a50-4775-81e3-d3ee113c7ec2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5\" (UID: \"ae881581-0a50-4775-81e3-d3ee113c7ec2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5" Oct 31 19:22:13.085447 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:13.085415 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zcmr\" (UniqueName: \"kubernetes.io/projected/ae881581-0a50-4775-81e3-d3ee113c7ec2-kube-api-access-4zcmr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5\" (UID: \"ae881581-0a50-4775-81e3-d3ee113c7ec2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5" Oct 31 19:22:13.143541 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:13.143506 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5" Oct 31 19:22:13.577903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:13.576914 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5"] Oct 31 19:22:13.581922 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:22:13.581892 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae881581_0a50_4775_81e3_d3ee113c7ec2.slice/crio-c3545889e23797028360f3b142c13d1cf98f11bcd6c15e6d2918cabec806bc88 WatchSource:0}: Error finding container c3545889e23797028360f3b142c13d1cf98f11bcd6c15e6d2918cabec806bc88: Status 404 returned error can't find the container with id c3545889e23797028360f3b142c13d1cf98f11bcd6c15e6d2918cabec806bc88 Oct 31 19:22:13.822336 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:13.822298 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5" event={"ID":"ae881581-0a50-4775-81e3-d3ee113c7ec2","Type":"ContainerStarted","Data":"c3545889e23797028360f3b142c13d1cf98f11bcd6c15e6d2918cabec806bc88"} Oct 31 19:22:15.623774 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.623738 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57"] Oct 31 19:22:15.626374 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.626356 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" Oct 31 19:22:15.678416 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.678383 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57"] Oct 31 19:22:15.722736 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.722705 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 19:22:15.722736 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.722705 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vkv8v\"" Oct 31 19:22:15.722944 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.722765 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 19:22:15.758217 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.758185 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-77g57\" (UID: \"edb5b2f3-16f7-48d4-96a5-16dd8fe35473\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" Oct 31 19:22:15.758393 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.758231 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvrm\" (UniqueName: \"kubernetes.io/projected/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-kube-api-access-npvrm\") pod \"kuadrant-operator-controller-manager-678d87766f-77g57\" (UID: \"edb5b2f3-16f7-48d4-96a5-16dd8fe35473\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" Oct 31 19:22:15.859672 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.859633 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-77g57\" (UID: \"edb5b2f3-16f7-48d4-96a5-16dd8fe35473\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" Oct 31 19:22:15.859842 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.859682 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvrm\" (UniqueName: \"kubernetes.io/projected/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-kube-api-access-npvrm\") pod \"kuadrant-operator-controller-manager-678d87766f-77g57\" (UID: \"edb5b2f3-16f7-48d4-96a5-16dd8fe35473\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" Oct 31 19:22:15.860139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.860111 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-77g57\" (UID: \"edb5b2f3-16f7-48d4-96a5-16dd8fe35473\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" Oct 31 19:22:15.932104 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.932038 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvrm\" (UniqueName: \"kubernetes.io/projected/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-kube-api-access-npvrm\") pod \"kuadrant-operator-controller-manager-678d87766f-77g57\" (UID: \"edb5b2f3-16f7-48d4-96a5-16dd8fe35473\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" Oct 31 19:22:15.939403 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:15.939378 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" Oct 31 19:22:16.465735 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:16.465606 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57"] Oct 31 19:22:16.469036 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:22:16.469000 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb5b2f3_16f7_48d4_96a5_16dd8fe35473.slice/crio-d67840b961a0a09eee02e86310a556b9593ff1c37cce280cda098c6ed2e4f2d0 WatchSource:0}: Error finding container d67840b961a0a09eee02e86310a556b9593ff1c37cce280cda098c6ed2e4f2d0: Status 404 returned error can't find the container with id d67840b961a0a09eee02e86310a556b9593ff1c37cce280cda098c6ed2e4f2d0 Oct 31 19:22:16.838043 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:16.838001 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" event={"ID":"edb5b2f3-16f7-48d4-96a5-16dd8fe35473","Type":"ContainerStarted","Data":"d67840b961a0a09eee02e86310a556b9593ff1c37cce280cda098c6ed2e4f2d0"} Oct 31 19:22:16.839627 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:16.839591 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5" event={"ID":"ae881581-0a50-4775-81e3-d3ee113c7ec2","Type":"ContainerStarted","Data":"376cce712b0d9c85ed83bf884c4df767d069ab906cc65c4697930026ee0b3b50"} Oct 31 19:22:16.955894 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:16.955817 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zmwb5" podStartSLOduration=1.983049415 podStartE2EDuration="4.955798341s" totalImagesPullingTime="2.972748926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:16.955112735 +0000 UTC m=+308.588506745" watchObservedRunningTime="2026-10-31 19:22:16.955798341 +0000 UTC m=+308.589192248" Oct 31 19:22:20.409228 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:22:20.409182 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:22:20.655344 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.655314 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw"] Oct 31 19:22:20.658091 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.658068 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" Oct 31 19:22:20.707377 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.707344 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 19:22:20.741077 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.741041 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6246eaed-3a6b-482b-a25d-fc111a84e89b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8bmvw\" (UID: \"6246eaed-3a6b-482b-a25d-fc111a84e89b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" Oct 31 19:22:20.741246 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.741102 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh99t\" (UniqueName: \"kubernetes.io/projected/6246eaed-3a6b-482b-a25d-fc111a84e89b-kube-api-access-zh99t\") pod \"cert-manager-webhook-5dbbbf75cc-8bmvw\" (UID: \"6246eaed-3a6b-482b-a25d-fc111a84e89b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" Oct 31 19:22:20.753648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.753604 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-96rxs\"" Oct 31 19:22:20.753788 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.753647 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 19:22:20.755277 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.755251 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw"] Oct 31 19:22:20.841819 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.841780 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6246eaed-3a6b-482b-a25d-fc111a84e89b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8bmvw\" (UID: \"6246eaed-3a6b-482b-a25d-fc111a84e89b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" Oct 31 19:22:20.842014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.841841 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh99t\" (UniqueName: \"kubernetes.io/projected/6246eaed-3a6b-482b-a25d-fc111a84e89b-kube-api-access-zh99t\") pod \"cert-manager-webhook-5dbbbf75cc-8bmvw\" (UID: \"6246eaed-3a6b-482b-a25d-fc111a84e89b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" Oct 31 19:22:20.899431 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.899394 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6246eaed-3a6b-482b-a25d-fc111a84e89b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8bmvw\" (UID: \"6246eaed-3a6b-482b-a25d-fc111a84e89b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" Oct 31 19:22:20.899574 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.899501 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh99t\" (UniqueName: \"kubernetes.io/projected/6246eaed-3a6b-482b-a25d-fc111a84e89b-kube-api-access-zh99t\") pod \"cert-manager-webhook-5dbbbf75cc-8bmvw\" (UID: \"6246eaed-3a6b-482b-a25d-fc111a84e89b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" Oct 31 19:22:20.970825 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:20.970796 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" Oct 31 19:22:21.033037 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.033010 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g"] Oct 31 19:22:21.040210 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.039877 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g" Oct 31 19:22:21.090475 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.090439 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g"] Oct 31 19:22:21.144929 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.144818 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9112bbf0-0381-4767-bf44-3f9548976432-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mm45g\" (UID: \"9112bbf0-0381-4767-bf44-3f9548976432\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g" Oct 31 19:22:21.144929 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.144863 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnr5\" (UniqueName: \"kubernetes.io/projected/9112bbf0-0381-4767-bf44-3f9548976432-kube-api-access-qcnr5\") pod \"cert-manager-cainjector-5c57bbd4fc-mm45g\" (UID: \"9112bbf0-0381-4767-bf44-3f9548976432\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g" Oct 31 19:22:21.246059 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.245988 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9112bbf0-0381-4767-bf44-3f9548976432-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mm45g\" (UID: \"9112bbf0-0381-4767-bf44-3f9548976432\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g" Oct 31 19:22:21.246059 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.246022 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnr5\" (UniqueName: \"kubernetes.io/projected/9112bbf0-0381-4767-bf44-3f9548976432-kube-api-access-qcnr5\") pod \"cert-manager-cainjector-5c57bbd4fc-mm45g\" (UID: \"9112bbf0-0381-4767-bf44-3f9548976432\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g" Oct 31 19:22:21.307116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.307081 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnr5\" (UniqueName: \"kubernetes.io/projected/9112bbf0-0381-4767-bf44-3f9548976432-kube-api-access-qcnr5\") pod \"cert-manager-cainjector-5c57bbd4fc-mm45g\" (UID: \"9112bbf0-0381-4767-bf44-3f9548976432\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g" Oct 31 19:22:21.319787 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.319762 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9112bbf0-0381-4767-bf44-3f9548976432-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mm45g\" (UID: \"9112bbf0-0381-4767-bf44-3f9548976432\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g" Oct 31 19:22:21.345931 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.345902 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw"] Oct 31 19:22:21.348245 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:22:21.348217 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6246eaed_3a6b_482b_a25d_fc111a84e89b.slice/crio-e089e904382d1d8401ff285d31f6c743ba583456806f92d47ec72f8ef6e55070 WatchSource:0}: Error finding container e089e904382d1d8401ff285d31f6c743ba583456806f92d47ec72f8ef6e55070: Status 404 returned error can't find the container with id e089e904382d1d8401ff285d31f6c743ba583456806f92d47ec72f8ef6e55070 Oct 31 19:22:21.354410 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.354386 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g" Oct 31 19:22:21.716783 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.716762 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g"] Oct 31 19:22:21.718947 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:22:21.718915 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9112bbf0_0381_4767_bf44_3f9548976432.slice/crio-3aa39de206c86cda875c23eb31ac222642b8112dfc4ae1eba514bcf28dc0b101 WatchSource:0}: Error finding container 3aa39de206c86cda875c23eb31ac222642b8112dfc4ae1eba514bcf28dc0b101: Status 404 returned error can't find the container with id 3aa39de206c86cda875c23eb31ac222642b8112dfc4ae1eba514bcf28dc0b101 Oct 31 19:22:21.867605 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.867571 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" event={"ID":"edb5b2f3-16f7-48d4-96a5-16dd8fe35473","Type":"ContainerStarted","Data":"0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5"} Oct 31 19:22:21.867763 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.867640 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" Oct 31 19:22:21.868781 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.868756 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g" event={"ID":"9112bbf0-0381-4767-bf44-3f9548976432","Type":"ContainerStarted","Data":"3aa39de206c86cda875c23eb31ac222642b8112dfc4ae1eba514bcf28dc0b101"} Oct 31 19:22:21.869901 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.869858 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" event={"ID":"6246eaed-3a6b-482b-a25d-fc111a84e89b","Type":"ContainerStarted","Data":"e089e904382d1d8401ff285d31f6c743ba583456806f92d47ec72f8ef6e55070"} Oct 31 19:22:21.979699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:21.979649 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" podStartSLOduration=2.013792424 podStartE2EDuration="6.979633638s" totalImagesPullingTime="4.965841214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:21.978249574 +0000 UTC m=+313.611643516" watchObservedRunningTime="2026-10-31 19:22:21.979633638 +0000 UTC m=+313.613027546" Oct 31 19:22:25.891128 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:25.891101 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" event={"ID":"6246eaed-3a6b-482b-a25d-fc111a84e89b","Type":"ContainerStarted","Data":"405bb6b78cc67fd53fa8287a46aa401d2c08f6958e741782c5be19618a2eef41"} Oct 31 19:22:25.891525 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:25.891220 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" Oct 31 19:22:25.998651 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:25.998602 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" podStartSLOduration=2.166826156 podStartE2EDuration="5.998587829s" totalImagesPullingTime="3.831761673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:25.998064305 +0000 UTC m=+317.631458214" watchObservedRunningTime="2026-10-31 19:22:25.998587829 +0000 UTC m=+317.631981736" Oct 31 19:22:26.897436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:26.897396 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g" event={"ID":"9112bbf0-0381-4767-bf44-3f9548976432","Type":"ContainerStarted","Data":"19d33d62a94b4adcb3546e041eb152f43ce7615dcfa15078ac9bf807ff293243"} Oct 31 19:22:27.005232 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:27.005184 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mm45g" podStartSLOduration=2.83976492 podStartE2EDuration="7.005169841s" totalImagesPullingTime="4.165404921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:27.004163008 +0000 UTC m=+318.637556915" watchObservedRunningTime="2026-10-31 19:22:27.005169841 +0000 UTC m=+318.638563748" Oct 31 19:22:30.001631 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.001597 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9"] Oct 31 19:22:30.004608 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.004588 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" Oct 31 19:22:30.053139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.053112 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 19:22:30.098690 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.098664 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 19:22:30.098690 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.098676 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 19:22:30.098924 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.098702 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7q2np\"" Oct 31 19:22:30.098924 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.098734 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 19:22:30.100430 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.100407 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9"] Oct 31 19:22:30.121384 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.121355 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a79481f4-b755-490e-8717-b542362b80ad-tmp\") pod \"kube-auth-proxy-9b655b5cf-k44r9\" (UID: \"a79481f4-b755-490e-8717-b542362b80ad\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" Oct 31 19:22:30.121520 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.121408 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhdr\" (UniqueName: \"kubernetes.io/projected/a79481f4-b755-490e-8717-b542362b80ad-kube-api-access-kbhdr\") pod \"kube-auth-proxy-9b655b5cf-k44r9\" (UID: \"a79481f4-b755-490e-8717-b542362b80ad\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" Oct 31 19:22:30.121566 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.121532 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79481f4-b755-490e-8717-b542362b80ad-tls-certs\") pod \"kube-auth-proxy-9b655b5cf-k44r9\" (UID: \"a79481f4-b755-490e-8717-b542362b80ad\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" Oct 31 19:22:30.222397 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.222364 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79481f4-b755-490e-8717-b542362b80ad-tls-certs\") pod \"kube-auth-proxy-9b655b5cf-k44r9\" (UID: \"a79481f4-b755-490e-8717-b542362b80ad\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" Oct 31 19:22:30.222397 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.222398 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a79481f4-b755-490e-8717-b542362b80ad-tmp\") pod \"kube-auth-proxy-9b655b5cf-k44r9\" (UID: \"a79481f4-b755-490e-8717-b542362b80ad\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" Oct 31 19:22:30.222628 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.222442 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhdr\" (UniqueName: \"kubernetes.io/projected/a79481f4-b755-490e-8717-b542362b80ad-kube-api-access-kbhdr\") pod \"kube-auth-proxy-9b655b5cf-k44r9\" (UID: \"a79481f4-b755-490e-8717-b542362b80ad\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" Oct 31 19:22:30.224790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.224761 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a79481f4-b755-490e-8717-b542362b80ad-tmp\") pod \"kube-auth-proxy-9b655b5cf-k44r9\" (UID: \"a79481f4-b755-490e-8717-b542362b80ad\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" Oct 31 19:22:30.224936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.224905 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79481f4-b755-490e-8717-b542362b80ad-tls-certs\") pod \"kube-auth-proxy-9b655b5cf-k44r9\" (UID: \"a79481f4-b755-490e-8717-b542362b80ad\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" Oct 31 19:22:30.277616 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.277549 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhdr\" (UniqueName: \"kubernetes.io/projected/a79481f4-b755-490e-8717-b542362b80ad-kube-api-access-kbhdr\") pod \"kube-auth-proxy-9b655b5cf-k44r9\" (UID: \"a79481f4-b755-490e-8717-b542362b80ad\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" Oct 31 19:22:30.315256 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.315230 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" Oct 31 19:22:30.677920 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.677835 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9"] Oct 31 19:22:30.680975 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:22:30.680943 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79481f4_b755_490e_8717_b542362b80ad.slice/crio-8055e402c410613c3693eeea74b631af98cffe80d0d75d10202db8898592c5e0 WatchSource:0}: Error finding container 8055e402c410613c3693eeea74b631af98cffe80d0d75d10202db8898592c5e0: Status 404 returned error can't find the container with id 8055e402c410613c3693eeea74b631af98cffe80d0d75d10202db8898592c5e0 Oct 31 19:22:30.916545 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:30.916510 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" event={"ID":"a79481f4-b755-490e-8717-b542362b80ad","Type":"ContainerStarted","Data":"8055e402c410613c3693eeea74b631af98cffe80d0d75d10202db8898592c5e0"} Oct 31 19:22:31.900247 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:31.900218 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" Oct 31 19:22:32.876653 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:32.876609 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" Oct 31 19:22:34.940175 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:34.940142 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" event={"ID":"a79481f4-b755-490e-8717-b542362b80ad","Type":"ContainerStarted","Data":"41f69e9663a47522a869a671e849226654bbea5a0c283417f1fce43ffb955c8b"} Oct 31 19:22:35.053287 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:22:35.053230 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9b655b5cf-k44r9" podStartSLOduration=2.390565258 podStartE2EDuration="6.053215837s" totalImagesPullingTime="3.662650579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:35.052235848 +0000 UTC m=+326.685629757" watchObservedRunningTime="2026-10-31 19:22:35.053215837 +0000 UTC m=+326.686609745" Oct 31 19:22:50.403006 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:22:50.402967 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:23:04.678778 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.678738 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h"] Oct 31 19:23:04.688844 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.688815 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:04.736097 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.736062 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h"] Oct 31 19:23:04.780527 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.780485 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 31 19:23:04.780712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.780591 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-krfpk\"" Oct 31 19:23:04.782780 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.782761 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 31 19:23:04.782926 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.782784 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 31 19:23:04.831572 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.831532 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dc842241-8a4d-4bb7-973c-27733403dcdd-manager-config\") pod \"jobset-controller-manager-774888cf88-wvg6h\" (UID: \"dc842241-8a4d-4bb7-973c-27733403dcdd\") " pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:04.831765 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.831584 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc842241-8a4d-4bb7-973c-27733403dcdd-metrics-certs\") pod \"jobset-controller-manager-774888cf88-wvg6h\" (UID: \"dc842241-8a4d-4bb7-973c-27733403dcdd\") " pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:04.831765 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.831757 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc842241-8a4d-4bb7-973c-27733403dcdd-cert\") pod \"jobset-controller-manager-774888cf88-wvg6h\" (UID: \"dc842241-8a4d-4bb7-973c-27733403dcdd\") " pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:04.831868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.831816 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hj5m\" (UniqueName: \"kubernetes.io/projected/dc842241-8a4d-4bb7-973c-27733403dcdd-kube-api-access-4hj5m\") pod \"jobset-controller-manager-774888cf88-wvg6h\" (UID: \"dc842241-8a4d-4bb7-973c-27733403dcdd\") " pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:04.933084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.932985 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc842241-8a4d-4bb7-973c-27733403dcdd-cert\") pod \"jobset-controller-manager-774888cf88-wvg6h\" (UID: \"dc842241-8a4d-4bb7-973c-27733403dcdd\") " pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:04.933084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.933034 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hj5m\" (UniqueName: \"kubernetes.io/projected/dc842241-8a4d-4bb7-973c-27733403dcdd-kube-api-access-4hj5m\") pod \"jobset-controller-manager-774888cf88-wvg6h\" (UID: \"dc842241-8a4d-4bb7-973c-27733403dcdd\") " pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:04.933084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.933061 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dc842241-8a4d-4bb7-973c-27733403dcdd-manager-config\") pod \"jobset-controller-manager-774888cf88-wvg6h\" (UID: \"dc842241-8a4d-4bb7-973c-27733403dcdd\") " pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:04.933379 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.933104 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc842241-8a4d-4bb7-973c-27733403dcdd-metrics-certs\") pod \"jobset-controller-manager-774888cf88-wvg6h\" (UID: \"dc842241-8a4d-4bb7-973c-27733403dcdd\") " pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:04.933781 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.933754 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dc842241-8a4d-4bb7-973c-27733403dcdd-manager-config\") pod \"jobset-controller-manager-774888cf88-wvg6h\" (UID: \"dc842241-8a4d-4bb7-973c-27733403dcdd\") " pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:04.935489 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.935466 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc842241-8a4d-4bb7-973c-27733403dcdd-cert\") pod \"jobset-controller-manager-774888cf88-wvg6h\" (UID: \"dc842241-8a4d-4bb7-973c-27733403dcdd\") " pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:04.935576 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.935507 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc842241-8a4d-4bb7-973c-27733403dcdd-metrics-certs\") pod \"jobset-controller-manager-774888cf88-wvg6h\" (UID: \"dc842241-8a4d-4bb7-973c-27733403dcdd\") " pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:04.992822 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:04.992793 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hj5m\" (UniqueName: \"kubernetes.io/projected/dc842241-8a4d-4bb7-973c-27733403dcdd-kube-api-access-4hj5m\") pod \"jobset-controller-manager-774888cf88-wvg6h\" (UID: \"dc842241-8a4d-4bb7-973c-27733403dcdd\") " pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:05.000629 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:05.000606 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:05.383999 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:05.383967 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h"] Oct 31 19:23:05.386480 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:23:05.386445 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc842241_8a4d_4bb7_973c_27733403dcdd.slice/crio-d5002a251a4a257d914df6668c85d05610a93d8d9b29493fa7bf13a37f376dab WatchSource:0}: Error finding container d5002a251a4a257d914df6668c85d05610a93d8d9b29493fa7bf13a37f376dab: Status 404 returned error can't find the container with id d5002a251a4a257d914df6668c85d05610a93d8d9b29493fa7bf13a37f376dab Oct 31 19:23:06.081767 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:06.081730 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" event={"ID":"dc842241-8a4d-4bb7-973c-27733403dcdd","Type":"ContainerStarted","Data":"d5002a251a4a257d914df6668c85d05610a93d8d9b29493fa7bf13a37f376dab"} Oct 31 19:23:10.101805 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:10.101772 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" event={"ID":"dc842241-8a4d-4bb7-973c-27733403dcdd","Type":"ContainerStarted","Data":"7fa699ab05789e2021709b1a92c0ce69601c36ae536399db5d6e8f7bcfac5f65"} Oct 31 19:23:10.102297 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:10.101844 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:10.208355 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:10.208290 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" podStartSLOduration=2.048867686 podStartE2EDuration="6.208272009s" totalImagesPullingTime="4.159404323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:10.207198322 +0000 UTC m=+361.840592231" watchObservedRunningTime="2026-10-31 19:23:10.208272009 +0000 UTC m=+361.841665917" Oct 31 19:23:20.401965 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:23:20.401864 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:23:21.111768 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:23:21.111740 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-774888cf88-wvg6h" Oct 31 19:23:50.402293 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:23:50.402253 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:24:20.401618 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:24:20.401574 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:24:50.401666 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:24:50.401573 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:25:20.401513 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:25:20.401477 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:25:50.402689 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:25:50.402641 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:26:20.402346 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:26:20.402266 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:26:50.402535 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:26:50.402494 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:27:20.402369 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:27:20.402331 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:27:50.402380 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:27:50.402282 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:28:20.401964 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:28:20.401930 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:28:50.401862 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:28:50.401820 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:29:20.402120 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:29:20.402037 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:29:50.401814 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:29:50.401777 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:30:20.401556 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:30:20.401521 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:30:50.401250 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:30:50.401176 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:31:20.401642 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:31:20.401590 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:31:50.401847 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:31:50.401813 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:32:20.401403 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:32:20.401324 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:32:48.238749 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.238719 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-77g57_edb5b2f3-16f7-48d4-96a5-16dd8fe35473/manager/0.log" Oct 31 19:32:48.637666 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.637574 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57"] Oct 31 19:32:48.637825 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.637802 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" podUID="edb5b2f3-16f7-48d4-96a5-16dd8fe35473" containerName="manager" containerID="cri-o://0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5" gracePeriod=10 Oct 31 19:32:48.638941 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.638919 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv"] Oct 31 19:32:48.642319 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.642301 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" Oct 31 19:32:48.701674 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.701645 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chf22\" (UniqueName: \"kubernetes.io/projected/19ec27d2-0211-40fa-a856-4a95d99c5209-kube-api-access-chf22\") pod \"kuadrant-operator-controller-manager-678d87766f-fvzlv\" (UID: \"19ec27d2-0211-40fa-a856-4a95d99c5209\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" Oct 31 19:32:48.701813 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.701685 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19ec27d2-0211-40fa-a856-4a95d99c5209-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fvzlv\" (UID: \"19ec27d2-0211-40fa-a856-4a95d99c5209\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" Oct 31 19:32:48.745062 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.745033 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv"] Oct 31 19:32:48.802716 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.802686 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chf22\" (UniqueName: \"kubernetes.io/projected/19ec27d2-0211-40fa-a856-4a95d99c5209-kube-api-access-chf22\") pod \"kuadrant-operator-controller-manager-678d87766f-fvzlv\" (UID: \"19ec27d2-0211-40fa-a856-4a95d99c5209\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" Oct 31 19:32:48.802909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.802736 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19ec27d2-0211-40fa-a856-4a95d99c5209-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fvzlv\" (UID: \"19ec27d2-0211-40fa-a856-4a95d99c5209\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" Oct 31 19:32:48.803086 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.803069 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19ec27d2-0211-40fa-a856-4a95d99c5209-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fvzlv\" (UID: \"19ec27d2-0211-40fa-a856-4a95d99c5209\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" Oct 31 19:32:48.858271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.858235 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chf22\" (UniqueName: \"kubernetes.io/projected/19ec27d2-0211-40fa-a856-4a95d99c5209-kube-api-access-chf22\") pod \"kuadrant-operator-controller-manager-678d87766f-fvzlv\" (UID: \"19ec27d2-0211-40fa-a856-4a95d99c5209\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" Oct 31 19:32:48.888232 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.888175 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" Oct 31 19:32:48.903309 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.903287 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-kube-api-access-npvrm\" (UniqueName: \"kubernetes.io/projected/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-kube-api-access-npvrm\") pod \"edb5b2f3-16f7-48d4-96a5-16dd8fe35473\" (UID: \"edb5b2f3-16f7-48d4-96a5-16dd8fe35473\") " Oct 31 19:32:48.903435 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.903330 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-extensions-socket-volume\") pod \"edb5b2f3-16f7-48d4-96a5-16dd8fe35473\" (UID: \"edb5b2f3-16f7-48d4-96a5-16dd8fe35473\") " Oct 31 19:32:48.903720 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.903697 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-extensions-socket-volume" pod "edb5b2f3-16f7-48d4-96a5-16dd8fe35473" (UID: "edb5b2f3-16f7-48d4-96a5-16dd8fe35473"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:32:48.905207 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:48.905178 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-kube-api-access-npvrm" pod "edb5b2f3-16f7-48d4-96a5-16dd8fe35473" (UID: "edb5b2f3-16f7-48d4-96a5-16dd8fe35473"). InnerVolumeSpecName "kube-api-access-npvrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:32:49.004692 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.004656 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npvrm\" (UniqueName: \"kubernetes.io/projected/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-kube-api-access-npvrm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:32:49.004692 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.004692 6107 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edb5b2f3-16f7-48d4-96a5-16dd8fe35473-extensions-socket-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:32:49.009529 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.009501 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" Oct 31 19:32:49.420032 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.419992 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv"] Oct 31 19:32:49.423094 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:32:49.423065 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ec27d2_0211_40fa_a856_4a95d99c5209.slice/crio-6df1253d3aa3da1561ea02989a4f88f757181c9dc157132669e5bbdbab6a27b5 WatchSource:0}: Error finding container 6df1253d3aa3da1561ea02989a4f88f757181c9dc157132669e5bbdbab6a27b5: Status 404 returned error can't find the container with id 6df1253d3aa3da1561ea02989a4f88f757181c9dc157132669e5bbdbab6a27b5 Oct 31 19:32:49.693276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.693184 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" event={"ID":"19ec27d2-0211-40fa-a856-4a95d99c5209","Type":"ContainerStarted","Data":"7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717"} Oct 31 19:32:49.693276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.693239 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" event={"ID":"19ec27d2-0211-40fa-a856-4a95d99c5209","Type":"ContainerStarted","Data":"6df1253d3aa3da1561ea02989a4f88f757181c9dc157132669e5bbdbab6a27b5"} Oct 31 19:32:49.693490 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.693291 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" Oct 31 19:32:49.694564 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.694537 6107 generic.go:356] "Generic (PLEG): container finished" podID="edb5b2f3-16f7-48d4-96a5-16dd8fe35473" containerID="0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5" exitCode=0 Oct 31 19:32:49.694685 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.694566 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" event={"ID":"edb5b2f3-16f7-48d4-96a5-16dd8fe35473","Type":"ContainerDied","Data":"0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5"} Oct 31 19:32:49.694685 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.694572 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" Oct 31 19:32:49.694685 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.694601 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57" event={"ID":"edb5b2f3-16f7-48d4-96a5-16dd8fe35473","Type":"ContainerDied","Data":"d67840b961a0a09eee02e86310a556b9593ff1c37cce280cda098c6ed2e4f2d0"} Oct 31 19:32:49.694685 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.694621 6107 scope.go:122] "RemoveContainer" containerID="0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5" Oct 31 19:32:49.704256 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.704240 6107 scope.go:122] "RemoveContainer" containerID="0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5" Oct 31 19:32:49.704550 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:32:49.704525 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5\": container with ID starting with 0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5 not found: ID does not exist" containerID="0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5" Oct 31 19:32:49.704642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.704556 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5"} err="failed to get container status \"0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5\": rpc error: code = NotFound desc = could not find container \"0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5\": container with ID starting with 0b6ba99c8620fbeca1e626d1b31e5871a58023fe0703a8f0b5c66bc3b027e7b5 not found: ID does not exist" Oct 31 19:32:49.814581 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.814533 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" podStartSLOduration=1.814520435 podStartE2EDuration="1.814520435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:49.812579821 +0000 UTC m=+941.445973720" watchObservedRunningTime="2026-10-31 19:32:49.814520435 +0000 UTC m=+941.447914342" Oct 31 19:32:49.917242 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.917207 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57"] Oct 31 19:32:49.930432 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:49.930404 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-77g57"] Oct 31 19:32:50.401757 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:32:50.401726 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:32:50.484686 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:32:50.484652 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edb5b2f3-16f7-48d4-96a5-16dd8fe35473" path="/var/lib/kubelet/pods/edb5b2f3-16f7-48d4-96a5-16dd8fe35473/volumes" Oct 31 19:33:00.702871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:33:00.702842 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" Oct 31 19:33:20.401680 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:33:20.401635 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:33:50.402091 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:33:50.402004 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:34:20.402247 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:34:20.402211 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:34:43.050844 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.050808 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77"] Oct 31 19:34:43.051306 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.051176 6107 cpu_manager.go:414] "containerMap: removing container" podUID="edb5b2f3-16f7-48d4-96a5-16dd8fe35473" containerName="manager" Oct 31 19:34:43.051306 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.051189 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb5b2f3-16f7-48d4-96a5-16dd8fe35473" containerName="manager" Oct 31 19:34:43.051306 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.051256 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb5b2f3-16f7-48d4-96a5-16dd8fe35473" containerName="manager" Oct 31 19:34:43.054435 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.054414 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.103597 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.103570 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 19:34:43.148915 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.148865 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-maas-api-tls\") pod \"maas-api-7d5f466c97-kqk77\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.149057 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.148934 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-metrics-certs\") pod \"maas-api-7d5f466c97-kqk77\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.149057 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.149004 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-openshift-service-ca\") pod \"maas-api-7d5f466c97-kqk77\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.149057 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.149045 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72fv2\" (UniqueName: \"kubernetes.io/projected/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-kube-api-access-72fv2\") pod \"maas-api-7d5f466c97-kqk77\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.149708 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.149692 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 19:34:43.149748 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.149724 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xd7wl\"" Oct 31 19:34:43.149748 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.149725 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 19:34:43.149814 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.149771 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 19:34:43.151488 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.151470 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77"] Oct 31 19:34:43.249871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.249841 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-metrics-certs\") pod \"maas-api-7d5f466c97-kqk77\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.250054 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.249903 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-openshift-service-ca\") pod \"maas-api-7d5f466c97-kqk77\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.250054 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.249947 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72fv2\" (UniqueName: \"kubernetes.io/projected/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-kube-api-access-72fv2\") pod \"maas-api-7d5f466c97-kqk77\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.250054 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.249974 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-maas-api-tls\") pod \"maas-api-7d5f466c97-kqk77\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.250731 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.250709 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-openshift-service-ca\") pod \"maas-api-7d5f466c97-kqk77\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.252612 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.252591 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-maas-api-tls\") pod \"maas-api-7d5f466c97-kqk77\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.252712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.252589 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-metrics-certs\") pod \"maas-api-7d5f466c97-kqk77\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.304230 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.304164 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72fv2\" (UniqueName: \"kubernetes.io/projected/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-kube-api-access-72fv2\") pod \"maas-api-7d5f466c97-kqk77\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.366226 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.366202 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:43.733327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.733299 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77"] Oct 31 19:34:43.736142 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:34:43.736110 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373c6bd5_cd00_43f7_b515_7aa8606a2bc6.slice/crio-767055c1235c013a6af4c7c1930c9fd7352331d30bba288821b20e67026d40ee WatchSource:0}: Error finding container 767055c1235c013a6af4c7c1930c9fd7352331d30bba288821b20e67026d40ee: Status 404 returned error can't find the container with id 767055c1235c013a6af4c7c1930c9fd7352331d30bba288821b20e67026d40ee Oct 31 19:34:43.737368 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:43.737352 6107 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:34:44.221061 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:44.221028 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" event={"ID":"373c6bd5-cd00-43f7-b515-7aa8606a2bc6","Type":"ContainerStarted","Data":"767055c1235c013a6af4c7c1930c9fd7352331d30bba288821b20e67026d40ee"} Oct 31 19:34:47.239639 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:47.239599 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" event={"ID":"373c6bd5-cd00-43f7-b515-7aa8606a2bc6","Type":"ContainerStarted","Data":"b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485"} Oct 31 19:34:47.240129 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:47.239728 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:34:47.346175 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:47.346121 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" podStartSLOduration=1.4334296389999999 podStartE2EDuration="4.34610513s" totalImagesPullingTime="2.912675491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:47.344701088 +0000 UTC m=+1058.978094996" watchObservedRunningTime="2026-10-31 19:34:47.34610513 +0000 UTC m=+1058.979499084" Oct 31 19:34:50.402684 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:34:50.402644 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:34:53.249744 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:34:53.249717 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 19:35:20.401402 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:35:20.401322 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:35:27.531646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.531611 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj"] Oct 31 19:35:27.534978 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.534959 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.584541 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.584513 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 19:35:27.640176 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.640140 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj"] Oct 31 19:35:27.662612 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.662583 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5cqm6\"" Oct 31 19:35:27.734022 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.733988 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.734181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.734028 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.734181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.734107 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.734181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.734140 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.734287 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.734213 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.734287 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.734238 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.734287 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.734262 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.734381 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.734317 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.734381 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.734373 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpp88\" (UniqueName: \"kubernetes.io/projected/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-kube-api-access-dpp88\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.835615 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.835539 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.835615 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.835571 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.835615 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.835597 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.835913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.835707 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.835913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.835754 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.835913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.835866 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.836099 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.835918 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.836099 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.836024 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.836099 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.836083 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.836247 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.836144 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpp88\" (UniqueName: \"kubernetes.io/projected/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-kube-api-access-dpp88\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.836247 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.836210 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.836349 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.836268 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.836423 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.836404 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.836472 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.836445 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.837956 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.837939 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.838312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.838292 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.892665 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.892638 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:27.892846 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:27.892823 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpp88\" (UniqueName: \"kubernetes.io/projected/dca75e6d-22eb-42ed-9d6c-4edb0d5678ef-kube-api-access-dpp88\") pod \"maas-default-gateway-openshift-default-58d898bd55-jkrrj\" (UID: \"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:28.146471 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:28.146384 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:28.557906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:28.557857 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj"] Oct 31 19:35:28.560634 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:35:28.560610 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca75e6d_22eb_42ed_9d6c_4edb0d5678ef.slice/crio-96ec33218295f3d52182e0d129a257a0312d9bd2326e4893521787e0ec2a3115 WatchSource:0}: Error finding container 96ec33218295f3d52182e0d129a257a0312d9bd2326e4893521787e0ec2a3115: Status 404 returned error can't find the container with id 96ec33218295f3d52182e0d129a257a0312d9bd2326e4893521787e0ec2a3115 Oct 31 19:35:29.429655 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:29.429619 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" event={"ID":"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef","Type":"ContainerStarted","Data":"96ec33218295f3d52182e0d129a257a0312d9bd2326e4893521787e0ec2a3115"} Oct 31 19:35:32.215324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:32.215296 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:35:32.215574 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:32.215360 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:35:32.215574 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:32.215391 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:35:32.449034 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:32.448989 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" event={"ID":"dca75e6d-22eb-42ed-9d6c-4edb0d5678ef","Type":"ContainerStarted","Data":"97a7e1578493084473bfffc6ff014674db7cd1cfb3bd509fbcbb555b3689e7a4"} Oct 31 19:35:32.605849 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:32.605787 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" podStartSLOduration=1.953829323 podStartE2EDuration="5.6057693s" totalImagesPullingTime="3.651939977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:32.605426605 +0000 UTC m=+1104.238820513" watchObservedRunningTime="2026-10-31 19:35:32.6057693 +0000 UTC m=+1104.239163211" Oct 31 19:35:33.147004 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:33.146957 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:33.148548 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:33.148519 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jkrrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 19:35:33.148677 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:33.148570 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" podUID="dca75e6d-22eb-42ed-9d6c-4edb0d5678ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 19:35:34.147477 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:34.147433 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jkrrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 19:35:34.147988 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:34.147500 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" podUID="dca75e6d-22eb-42ed-9d6c-4edb0d5678ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 19:35:35.147109 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:35.147071 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jkrrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 19:35:35.147273 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:35.147128 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" podUID="dca75e6d-22eb-42ed-9d6c-4edb0d5678ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 19:35:36.147247 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:36.147208 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jkrrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 19:35:36.147735 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:36.147280 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" podUID="dca75e6d-22eb-42ed-9d6c-4edb0d5678ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 19:35:37.147705 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:37.147666 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jkrrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 19:35:37.148122 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:37.147722 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" podUID="dca75e6d-22eb-42ed-9d6c-4edb0d5678ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 19:35:38.151214 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:38.151183 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:38.151605 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:38.151487 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:38.152240 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:38.152221 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" Oct 31 19:35:42.576895 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.576856 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h"] Oct 31 19:35:42.580500 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.580481 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.591103 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.591058 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4"] Oct 31 19:35:42.594517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.594501 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.636129 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.636102 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h"] Oct 31 19:35:42.687862 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.687829 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.687862 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.687869 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.688090 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.687905 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.688090 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.687988 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.688186 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688116 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba362a7-7884-4660-b218-98c97194513a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.688222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688182 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba362a7-7884-4660-b218-98c97194513a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.688222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688212 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.688316 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688235 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.688372 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688356 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.688412 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688388 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.688445 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688432 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.688530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688508 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.688589 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688545 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.688589 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688574 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.688654 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688631 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.688686 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688661 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlptc\" (UniqueName: \"kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-kube-api-access-xlptc\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.688724 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688712 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.688773 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.688754 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmpzp\" (UniqueName: \"kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-kube-api-access-qmpzp\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.740783 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.740751 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4"] Oct 31 19:35:42.789654 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.789620 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.789858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.789663 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmpzp\" (UniqueName: \"kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-kube-api-access-qmpzp\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.789858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.789709 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.789858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.789735 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.789858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.789764 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.789858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.789787 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.789858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.789817 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba362a7-7884-4660-b218-98c97194513a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.790209 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790002 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba362a7-7884-4660-b218-98c97194513a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.790209 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790053 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.790209 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790077 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.790209 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790110 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.790209 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790113 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.790209 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790137 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.790209 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790160 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.790209 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790167 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.790209 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790192 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.790209 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790195 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.790689 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790236 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.790689 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790272 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.790689 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790309 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.790689 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790338 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlptc\" (UniqueName: \"kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-kube-api-access-xlptc\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.790689 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790428 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.790689 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790486 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.791019 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790740 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.791019 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790764 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba362a7-7884-4660-b218-98c97194513a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.791019 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.790766 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.791019 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.791004 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.791211 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.791086 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.792360 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.792329 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.792573 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.792548 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba362a7-7884-4660-b218-98c97194513a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.792697 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.792639 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.792759 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.792728 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.842930 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.842839 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.842930 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.842848 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.842930 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.842915 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmpzp\" (UniqueName: \"kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-kube-api-access-qmpzp\") pod \"maas-default-gateway-openshift-default-58d898bd55-tr92h\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.843108 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.842949 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlptc\" (UniqueName: \"kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-kube-api-access-xlptc\") pod \"maas-default-gateway-openshift-default-58d898bd55-blhc4\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:42.892468 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.892446 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:42.905739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:42.905715 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:43.306263 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.306237 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h"] Oct 31 19:35:43.308161 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.308140 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4"] Oct 31 19:35:43.309233 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:35:43.309210 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe6fc2e_343d_493f_9f8f_d3db1343e86b.slice/crio-4500efc6f17baf653393774285528e4ef2645087e8a2d5c14a853819215db67e WatchSource:0}: Error finding container 4500efc6f17baf653393774285528e4ef2645087e8a2d5c14a853819215db67e: Status 404 returned error can't find the container with id 4500efc6f17baf653393774285528e4ef2645087e8a2d5c14a853819215db67e Oct 31 19:35:43.310602 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:35:43.310576 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba362a7_7884_4660_b218_98c97194513a.slice/crio-def36d9b87a27fc4a1d3ab908ded85eebe29a336fb75370e6d27d4472efedee9 WatchSource:0}: Error finding container def36d9b87a27fc4a1d3ab908ded85eebe29a336fb75370e6d27d4472efedee9: Status 404 returned error can't find the container with id def36d9b87a27fc4a1d3ab908ded85eebe29a336fb75370e6d27d4472efedee9 Oct 31 19:35:43.311475 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.311438 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:35:43.311599 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.311514 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:35:43.311599 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.311546 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:35:43.312565 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.312541 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:35:43.312640 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.312593 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:35:43.312640 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.312623 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:35:43.515933 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.515872 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" event={"ID":"9fe6fc2e-343d-493f-9f8f-d3db1343e86b","Type":"ContainerStarted","Data":"13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436"} Oct 31 19:35:43.516220 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.516009 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" event={"ID":"9fe6fc2e-343d-493f-9f8f-d3db1343e86b","Type":"ContainerStarted","Data":"4500efc6f17baf653393774285528e4ef2645087e8a2d5c14a853819215db67e"} Oct 31 19:35:43.517765 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.517739 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" event={"ID":"cba362a7-7884-4660-b218-98c97194513a","Type":"ContainerStarted","Data":"2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85"} Oct 31 19:35:43.517910 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.517785 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" event={"ID":"cba362a7-7884-4660-b218-98c97194513a","Type":"ContainerStarted","Data":"def36d9b87a27fc4a1d3ab908ded85eebe29a336fb75370e6d27d4472efedee9"} Oct 31 19:35:43.626137 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.626034 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" podStartSLOduration=1.626016629 podStartE2EDuration="1.626016629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:43.624459712 +0000 UTC m=+1115.257853620" watchObservedRunningTime="2026-10-31 19:35:43.626016629 +0000 UTC m=+1115.259410538" Oct 31 19:35:43.893016 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.892941 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:43.908041 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:43.907999 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:44.894901 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:44.894850 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tr92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 19:35:44.895321 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:44.894941 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" podUID="9fe6fc2e-343d-493f-9f8f-d3db1343e86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 19:35:44.920906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:44.919304 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 19:35:44.920906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:44.919372 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" podUID="cba362a7-7884-4660-b218-98c97194513a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 19:35:45.893547 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:45.893510 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tr92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 19:35:45.893728 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:45.893565 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" podUID="9fe6fc2e-343d-493f-9f8f-d3db1343e86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 19:35:45.915906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:45.914280 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 19:35:45.915906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:45.914344 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" podUID="cba362a7-7884-4660-b218-98c97194513a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 19:35:46.893036 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:46.893008 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tr92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 19:35:46.893284 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:46.893255 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" podUID="9fe6fc2e-343d-493f-9f8f-d3db1343e86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 19:35:46.906913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:46.906198 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 19:35:46.906913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:46.906252 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" podUID="cba362a7-7884-4660-b218-98c97194513a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 19:35:47.893413 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:47.893380 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tr92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 19:35:47.893916 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:47.893436 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" podUID="9fe6fc2e-343d-493f-9f8f-d3db1343e86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 19:35:47.906693 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:47.906663 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 19:35:47.913001 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:47.912966 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" podUID="cba362a7-7884-4660-b218-98c97194513a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 19:35:48.893137 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:48.893109 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tr92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 31 19:35:48.893322 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:48.893155 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" podUID="9fe6fc2e-343d-493f-9f8f-d3db1343e86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 31 19:35:48.944355 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:48.928491 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-blhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 19:35:48.944355 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:48.928562 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" podUID="cba362a7-7884-4660-b218-98c97194513a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 19:35:49.896778 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:49.896745 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:49.897122 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:49.897098 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:49.897840 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:49.897824 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:35:49.909411 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:49.909389 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:49.909666 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:49.909652 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:49.910241 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:49.910222 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:35:50.051659 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:50.051600 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" podStartSLOduration=8.051587032 podStartE2EDuration="8.051587032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:43.737170578 +0000 UTC m=+1115.370564486" watchObservedRunningTime="2026-10-31 19:35:50.051587032 +0000 UTC m=+1121.684980939" Oct 31 19:35:50.402563 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:35:50.402531 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:35:52.090359 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.090326 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz"] Oct 31 19:35:52.094911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.094873 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.148571 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.148536 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz"] Oct 31 19:35:52.192066 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.192035 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 19:35:52.192066 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.192032 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 19:35:52.192310 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.192033 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l8x7h\"" Oct 31 19:35:52.192310 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.192069 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 31 19:35:52.213505 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.213477 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ljj\" (UniqueName: \"kubernetes.io/projected/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-kube-api-access-p9ljj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.213647 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.213523 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.213705 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.213661 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.213755 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.213711 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.213755 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.213727 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.213852 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.213818 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.314516 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.314478 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.314716 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.314526 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.314716 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.314541 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.314716 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.314558 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.314716 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.314683 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ljj\" (UniqueName: \"kubernetes.io/projected/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-kube-api-access-p9ljj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.314961 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.314752 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.315065 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.315043 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.315118 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.315063 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.315172 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.315155 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.316847 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.316827 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.317039 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.317023 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.368292 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.368221 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ljj\" (UniqueName: \"kubernetes.io/projected/a78e6484-6c0b-4c24-a693-1b7c55fe70f3-kube-api-access-p9ljj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-shbmz\" (UID: \"a78e6484-6c0b-4c24-a693-1b7c55fe70f3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.407358 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.407330 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:35:52.770307 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:52.770283 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz"] Oct 31 19:35:52.771868 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:35:52.771836 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78e6484_6c0b_4c24_a693_1b7c55fe70f3.slice/crio-4fe770124d859055c40411c1d44561348c4997ddb36641b0851fc83a0c95a343 WatchSource:0}: Error finding container 4fe770124d859055c40411c1d44561348c4997ddb36641b0851fc83a0c95a343: Status 404 returned error can't find the container with id 4fe770124d859055c40411c1d44561348c4997ddb36641b0851fc83a0c95a343 Oct 31 19:35:53.595511 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:53.595473 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" event={"ID":"a78e6484-6c0b-4c24-a693-1b7c55fe70f3","Type":"ContainerStarted","Data":"4fe770124d859055c40411c1d44561348c4997ddb36641b0851fc83a0c95a343"} Oct 31 19:35:54.627872 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:54.627784 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" event={"ID":"a78e6484-6c0b-4c24-a693-1b7c55fe70f3","Type":"ContainerStarted","Data":"b6b3bc3575cf7940204c50b2347dceab614368f798d01f475b7043a8e2574516"} Oct 31 19:35:55.407843 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:35:55.407811 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:36:01.813404 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:01.813341 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" podStartSLOduration=8.256503038 podStartE2EDuration="9.81332399s" totalImagesPullingTime="1.556820952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:54.747207528 +0000 UTC m=+1126.380601437" watchObservedRunningTime="2026-10-31 19:36:01.81332399 +0000 UTC m=+1133.446717898" Oct 31 19:36:01.816663 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:01.816621 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6m88h"] Oct 31 19:36:01.834016 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:01.833991 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:01.877222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:01.876824 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6m88h"] Oct 31 19:36:01.935138 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:01.935106 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 19:36:01.949911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:01.944335 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:01.950217 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:01.950188 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:01.952904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:01.950335 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtjhd\" (UniqueName: \"kubernetes.io/projected/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-kube-api-access-xtjhd\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:01.958044 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:01.958016 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:01.958220 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:01.958203 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:01.958355 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:01.958339 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-home\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.063579 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.063481 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.064290 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.064268 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.064827 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.064808 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-home\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.065421 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.065403 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.065997 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.065980 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.066433 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.066414 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtjhd\" (UniqueName: \"kubernetes.io/projected/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-kube-api-access-xtjhd\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.066894 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.066353 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.067455 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.064195 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.067455 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.065213 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-home\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.067455 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.067408 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.070318 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.070294 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.138791 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.138721 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtjhd\" (UniqueName: \"kubernetes.io/projected/3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278-kube-api-access-xtjhd\") pod \"echo-simulator-kserve-64b6f8f96c-6m88h\" (UID: \"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.177217 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.177142 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:02.408280 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.408194 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:36:02.420505 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.420476 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:36:02.633277 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.633253 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6m88h"] Oct 31 19:36:02.635654 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:36:02.635622 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be4fac3_79cf_4dfc_bf3b_6ff4f8d60278.slice/crio-6c5a8152b7db277c2fe4777787c6e563088b3a9798d41387bf65b5519bcbb8d3 WatchSource:0}: Error finding container 6c5a8152b7db277c2fe4777787c6e563088b3a9798d41387bf65b5519bcbb8d3: Status 404 returned error can't find the container with id 6c5a8152b7db277c2fe4777787c6e563088b3a9798d41387bf65b5519bcbb8d3 Oct 31 19:36:02.673982 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.673904 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" event={"ID":"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278","Type":"ContainerStarted","Data":"6c5a8152b7db277c2fe4777787c6e563088b3a9798d41387bf65b5519bcbb8d3"} Oct 31 19:36:02.684942 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:02.684919 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" Oct 31 19:36:05.692223 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:05.692125 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" event={"ID":"3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278","Type":"ContainerStarted","Data":"8f46300008bf1224021ca3029fd8db5dcdcc4a814b065e46a5d6ca9318defa9c"} Oct 31 19:36:05.801427 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:05.801372 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" podStartSLOduration=2.174410639 podStartE2EDuration="4.80135592s" totalImagesPullingTime="2.626945281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:05.799537953 +0000 UTC m=+1137.432931861" watchObservedRunningTime="2026-10-31 19:36:05.80135592 +0000 UTC m=+1137.434749875" Oct 31 19:36:06.178295 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:06.178262 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:10.304478 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.304446 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v"] Oct 31 19:36:10.307078 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.307062 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.362354 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.362317 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5019f17d-138f-4f54-b963-51645b530ffe-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.362540 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.362394 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5019f17d-138f-4f54-b963-51645b530ffe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.362540 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.362501 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5019f17d-138f-4f54-b963-51645b530ffe-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.362643 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.362540 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9f9n\" (UniqueName: \"kubernetes.io/projected/5019f17d-138f-4f54-b963-51645b530ffe-kube-api-access-n9f9n\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.362643 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.362564 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5019f17d-138f-4f54-b963-51645b530ffe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.362643 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.362589 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5019f17d-138f-4f54-b963-51645b530ffe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.401361 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.401332 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 19:36:10.403113 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.403084 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v"] Oct 31 19:36:10.463562 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.463525 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5019f17d-138f-4f54-b963-51645b530ffe-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.463740 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.463575 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9f9n\" (UniqueName: \"kubernetes.io/projected/5019f17d-138f-4f54-b963-51645b530ffe-kube-api-access-n9f9n\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.463740 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.463692 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5019f17d-138f-4f54-b963-51645b530ffe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.463740 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.463726 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5019f17d-138f-4f54-b963-51645b530ffe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.463870 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.463799 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5019f17d-138f-4f54-b963-51645b530ffe-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.463946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.463918 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5019f17d-138f-4f54-b963-51645b530ffe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.464109 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.464088 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5019f17d-138f-4f54-b963-51645b530ffe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.464237 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.464204 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5019f17d-138f-4f54-b963-51645b530ffe-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.464296 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.464277 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5019f17d-138f-4f54-b963-51645b530ffe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.465747 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.465724 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5019f17d-138f-4f54-b963-51645b530ffe-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.466114 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.466097 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5019f17d-138f-4f54-b963-51645b530ffe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.518360 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.518332 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9f9n\" (UniqueName: \"kubernetes.io/projected/5019f17d-138f-4f54-b963-51645b530ffe-kube-api-access-n9f9n\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v\" (UID: \"5019f17d-138f-4f54-b963-51645b530ffe\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.617574 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.617494 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:10.986824 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:10.986795 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v"] Oct 31 19:36:10.989256 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:36:10.989227 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5019f17d_138f_4f54_b963_51645b530ffe.slice/crio-a6d2c250de7e97f437b5fd78274e047344f2b594ce99b69e17c38573be9bd409 WatchSource:0}: Error finding container a6d2c250de7e97f437b5fd78274e047344f2b594ce99b69e17c38573be9bd409: Status 404 returned error can't find the container with id a6d2c250de7e97f437b5fd78274e047344f2b594ce99b69e17c38573be9bd409 Oct 31 19:36:11.733661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:11.733624 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" event={"ID":"5019f17d-138f-4f54-b963-51645b530ffe","Type":"ContainerStarted","Data":"ac625e61b4ba879e134a121f37e146ee774123ac4e8b4f0fd2f64eb9fb489394"} Oct 31 19:36:11.733661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:11.733665 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" event={"ID":"5019f17d-138f-4f54-b963-51645b530ffe","Type":"ContainerStarted","Data":"a6d2c250de7e97f437b5fd78274e047344f2b594ce99b69e17c38573be9bd409"} Oct 31 19:36:11.842451 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:11.842344 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podStartSLOduration=1.301380357 podStartE2EDuration="1.842327614s" totalImagesPullingTime="540.947257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:11.840753396 +0000 UTC m=+1143.474147303" watchObservedRunningTime="2026-10-31 19:36:11.842327614 +0000 UTC m=+1143.475721555" Oct 31 19:36:12.178173 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:12.178075 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:12.190128 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:12.190095 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:12.618478 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:12.618441 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:12.749750 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:12.749724 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6m88h" Oct 31 19:36:19.276151 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.276116 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm"] Oct 31 19:36:19.278688 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.278669 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.354544 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.354508 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 31 19:36:19.356925 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.356899 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d359d1-caca-4b2d-9a96-a7ce1363a407-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.357073 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.356936 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d359d1-caca-4b2d-9a96-a7ce1363a407-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.357073 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.357054 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d359d1-caca-4b2d-9a96-a7ce1363a407-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.357190 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.357085 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txlq\" (UniqueName: \"kubernetes.io/projected/b0d359d1-caca-4b2d-9a96-a7ce1363a407-kube-api-access-2txlq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.357190 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.357118 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d359d1-caca-4b2d-9a96-a7ce1363a407-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.357298 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.357202 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d359d1-caca-4b2d-9a96-a7ce1363a407-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.375018 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.374992 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm"] Oct 31 19:36:19.458725 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.458678 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d359d1-caca-4b2d-9a96-a7ce1363a407-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.458937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.458741 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txlq\" (UniqueName: \"kubernetes.io/projected/b0d359d1-caca-4b2d-9a96-a7ce1363a407-kube-api-access-2txlq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.458937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.458774 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d359d1-caca-4b2d-9a96-a7ce1363a407-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.458937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.458819 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d359d1-caca-4b2d-9a96-a7ce1363a407-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.458937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.458866 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d359d1-caca-4b2d-9a96-a7ce1363a407-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.458937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.458929 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d359d1-caca-4b2d-9a96-a7ce1363a407-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.459210 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.459192 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d359d1-caca-4b2d-9a96-a7ce1363a407-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.459304 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.459281 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d359d1-caca-4b2d-9a96-a7ce1363a407-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.459464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.459441 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d359d1-caca-4b2d-9a96-a7ce1363a407-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.461113 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.461087 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d359d1-caca-4b2d-9a96-a7ce1363a407-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.461379 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.461358 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d359d1-caca-4b2d-9a96-a7ce1363a407-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.513014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.512976 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txlq\" (UniqueName: \"kubernetes.io/projected/b0d359d1-caca-4b2d-9a96-a7ce1363a407-kube-api-access-2txlq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-r5tcm\" (UID: \"b0d359d1-caca-4b2d-9a96-a7ce1363a407\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.590826 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.590741 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:19.953762 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:19.953737 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm"] Oct 31 19:36:19.956755 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:36:19.956726 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d359d1_caca_4b2d_9a96_a7ce1363a407.slice/crio-a10b45ae740ce4f48feb109e4b1ab884e50ba0c315c52eb2f830f9b5b31a89a4 WatchSource:0}: Error finding container a10b45ae740ce4f48feb109e4b1ab884e50ba0c315c52eb2f830f9b5b31a89a4: Status 404 returned error can't find the container with id a10b45ae740ce4f48feb109e4b1ab884e50ba0c315c52eb2f830f9b5b31a89a4 Oct 31 19:36:20.403597 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:36:20.403560 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:36:20.618199 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:20.618141 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:20.631925 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:20.631866 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:20.811808 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:20.811770 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" event={"ID":"b0d359d1-caca-4b2d-9a96-a7ce1363a407","Type":"ContainerStarted","Data":"f22b8e65a4cfa33ee7aeac3a6b3464f1b6000203a605bf14ecfa526c32f05ece"} Oct 31 19:36:20.812073 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:20.811838 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" event={"ID":"b0d359d1-caca-4b2d-9a96-a7ce1363a407","Type":"ContainerStarted","Data":"a10b45ae740ce4f48feb109e4b1ab884e50ba0c315c52eb2f830f9b5b31a89a4"} Oct 31 19:36:20.830265 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:20.830230 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:36:20.923169 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:20.923088 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" podStartSLOduration=1.365539416 podStartE2EDuration="1.923067662s" totalImagesPullingTime="557.528246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:20.922823863 +0000 UTC m=+1152.556217771" watchObservedRunningTime="2026-10-31 19:36:20.923067662 +0000 UTC m=+1152.556461588" Oct 31 19:36:21.591553 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:21.591521 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:29.591459 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:29.591412 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:29.603454 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:29.603423 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:29.893020 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:29.892944 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-r5tcm" Oct 31 19:36:50.403368 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:36:50.403264 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:36:58.312528 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.312490 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-8584f67695-vgddt"] Oct 31 19:36:58.316014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.315994 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" Oct 31 19:36:58.369266 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.369234 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-8584f67695-vgddt"] Oct 31 19:36:58.412441 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.412396 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 31 19:36:58.412441 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.412417 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 19:36:58.412441 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.412410 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 31 19:36:58.412441 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.412442 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-9zrdb\"" Oct 31 19:36:58.412742 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.412473 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 19:36:58.473707 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.473672 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjr69\" (UniqueName: \"kubernetes.io/projected/2b9d3a1a-82a2-491a-a272-0275351a3281-kube-api-access-jjr69\") pod \"ai-gateway-controller-8584f67695-vgddt\" (UID: \"2b9d3a1a-82a2-491a-a272-0275351a3281\") " pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" Oct 31 19:36:58.473909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.473797 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9d3a1a-82a2-491a-a272-0275351a3281-metrics-certs\") pod \"ai-gateway-controller-8584f67695-vgddt\" (UID: \"2b9d3a1a-82a2-491a-a272-0275351a3281\") " pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" Oct 31 19:36:58.574645 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.574544 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9d3a1a-82a2-491a-a272-0275351a3281-metrics-certs\") pod \"ai-gateway-controller-8584f67695-vgddt\" (UID: \"2b9d3a1a-82a2-491a-a272-0275351a3281\") " pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" Oct 31 19:36:58.574846 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.574817 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjr69\" (UniqueName: \"kubernetes.io/projected/2b9d3a1a-82a2-491a-a272-0275351a3281-kube-api-access-jjr69\") pod \"ai-gateway-controller-8584f67695-vgddt\" (UID: \"2b9d3a1a-82a2-491a-a272-0275351a3281\") " pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" Oct 31 19:36:58.577068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.577038 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b9d3a1a-82a2-491a-a272-0275351a3281-metrics-certs\") pod \"ai-gateway-controller-8584f67695-vgddt\" (UID: \"2b9d3a1a-82a2-491a-a272-0275351a3281\") " pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" Oct 31 19:36:58.628163 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.628131 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjr69\" (UniqueName: \"kubernetes.io/projected/2b9d3a1a-82a2-491a-a272-0275351a3281-kube-api-access-jjr69\") pod \"ai-gateway-controller-8584f67695-vgddt\" (UID: \"2b9d3a1a-82a2-491a-a272-0275351a3281\") " pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" Oct 31 19:36:58.628332 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.628316 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" Oct 31 19:36:58.991576 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:58.991547 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-8584f67695-vgddt"] Oct 31 19:36:58.993709 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:36:58.993679 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9d3a1a_82a2_491a_a272_0275351a3281.slice/crio-a085f0a6f0079e14c7f1d0ee980c9101ea09393bc78f94c981459812bfb17536 WatchSource:0}: Error finding container a085f0a6f0079e14c7f1d0ee980c9101ea09393bc78f94c981459812bfb17536: Status 404 returned error can't find the container with id a085f0a6f0079e14c7f1d0ee980c9101ea09393bc78f94c981459812bfb17536 Oct 31 19:36:59.027963 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:36:59.027925 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" event={"ID":"2b9d3a1a-82a2-491a-a272-0275351a3281","Type":"ContainerStarted","Data":"a085f0a6f0079e14c7f1d0ee980c9101ea09393bc78f94c981459812bfb17536"} Oct 31 19:37:02.045134 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:02.045092 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" event={"ID":"2b9d3a1a-82a2-491a-a272-0275351a3281","Type":"ContainerStarted","Data":"83c39c559fc00531a736d66cbca139bb6bfeeae3f98985e66c462ffc422618ac"} Oct 31 19:37:02.045643 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:02.045187 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" Oct 31 19:37:02.156501 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:02.156438 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" podStartSLOduration=1.936062019 podStartE2EDuration="4.15642315s" totalImagesPullingTime="2.220361131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:02.154477819 +0000 UTC m=+1193.787871731" watchObservedRunningTime="2026-10-31 19:37:02.15642315 +0000 UTC m=+1193.789817057" Oct 31 19:37:13.052601 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:13.052571 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-8584f67695-vgddt" Oct 31 19:37:20.403009 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:37:20.402970 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:37:43.783941 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.783906 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk"] Oct 31 19:37:43.787538 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.787517 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.862653 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.862617 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk"] Oct 31 19:37:43.881391 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.881365 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b5155c-openshift-default-dockercfg-d5fhs\"" Oct 31 19:37:43.894951 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.894920 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.895251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.894982 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.895251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.895069 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.895251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.895143 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.895251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.895175 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf8701ec-59c6-45e1-a62f-f50d102f9758-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.895251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.895229 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.897025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.895260 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.897025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.895313 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq27c\" (UniqueName: \"kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-kube-api-access-pq27c\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.897025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.895519 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.996384 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.996351 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.996623 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.996397 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.996623 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.996414 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf8701ec-59c6-45e1-a62f-f50d102f9758-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.996623 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.996450 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.996623 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.996466 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.996623 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.996500 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq27c\" (UniqueName: \"kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-kube-api-access-pq27c\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.996623 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.996569 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.996623 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.996601 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.997032 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.996646 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.997032 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.997001 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.997137 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.997116 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.997191 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.997177 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.997251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.997197 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.997404 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.997381 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf8701ec-59c6-45e1-a62f-f50d102f9758-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.998823 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.998802 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:43.999100 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:43.999083 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:44.048755 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.048694 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:44.048953 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.048932 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq27c\" (UniqueName: \"kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-kube-api-access-pq27c\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:44.091073 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.091046 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch"] Oct 31 19:37:44.095567 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.095549 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.099874 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.099852 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:44.143972 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.143932 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-570915-openshift-default-dockercfg-87zft\"" Oct 31 19:37:44.150307 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.150277 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch"] Oct 31 19:37:44.198641 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.198595 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.198784 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.198646 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.198784 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.198769 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.198858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.198809 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.198858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.198838 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.198974 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.198952 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.199008 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.198989 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8qj6\" (UniqueName: \"kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-kube-api-access-c8qj6\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.199073 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.199060 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.199124 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.199112 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.300277 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300195 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.300277 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300238 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.300277 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300253 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.300591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300294 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.300591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300317 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.300591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300338 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.300591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300362 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.300591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300388 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8qj6\" (UniqueName: \"kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-kube-api-access-c8qj6\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.300591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300428 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.300862 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300694 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.300862 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300754 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.300862 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300780 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.301005 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.300936 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.301191 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.301173 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.303092 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.303073 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.303333 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.303316 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.353729 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.353703 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.353876 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.353860 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8qj6\" (UniqueName: \"kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-kube-api-access-c8qj6\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.409690 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.409661 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:44.517932 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.517904 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk"] Oct 31 19:37:44.520009 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:37:44.519982 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8701ec_59c6_45e1_a62f_f50d102f9758.slice/crio-f5bc2f55a18f73f9b6d05ef6aa1de69c544d589bfd0dd7b4b5f9ffe9ce29a5b3 WatchSource:0}: Error finding container f5bc2f55a18f73f9b6d05ef6aa1de69c544d589bfd0dd7b4b5f9ffe9ce29a5b3: Status 404 returned error can't find the container with id f5bc2f55a18f73f9b6d05ef6aa1de69c544d589bfd0dd7b4b5f9ffe9ce29a5b3 Oct 31 19:37:44.522442 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.522411 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:37:44.522555 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.522483 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:37:44.522555 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.522527 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:37:44.786511 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.786486 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch"] Oct 31 19:37:44.788635 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:37:44.788604 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9aad8c_a2ad_4989_a088_4cda2f8a2a68.slice/crio-75f985a13dbc2318a59422de867e72919b697fbef4d193fcd28555b88b9b8254 WatchSource:0}: Error finding container 75f985a13dbc2318a59422de867e72919b697fbef4d193fcd28555b88b9b8254: Status 404 returned error can't find the container with id 75f985a13dbc2318a59422de867e72919b697fbef4d193fcd28555b88b9b8254 Oct 31 19:37:44.791036 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.791000 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:37:44.791147 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.791079 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:37:44.791147 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:44.791121 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:37:45.004917 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.004862 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl"] Oct 31 19:37:45.009145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.009123 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.064430 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.064397 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl"] Oct 31 19:37:45.065431 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.065409 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-662be1-openshift-default-dockercfg-bwkr4\"" Oct 31 19:37:45.110246 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.110161 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.110246 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.110205 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704ab7da-672d-4ed0-8efa-7177dde795b0-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.110449 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.110288 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.110449 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.110325 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.110449 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.110397 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.110619 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.110485 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.110619 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.110514 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9mg7\" (UniqueName: \"kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-kube-api-access-t9mg7\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.110710 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.110625 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.110747 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.110721 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.212334 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.212288 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.212334 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.212336 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.212596 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.212360 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mg7\" (UniqueName: \"kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-kube-api-access-t9mg7\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.212596 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.212423 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.212596 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.212496 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.212596 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.212538 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.212596 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.212571 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704ab7da-672d-4ed0-8efa-7177dde795b0-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.212833 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.212600 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.212833 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.212633 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.212833 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.212707 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.213012 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.212867 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.213012 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.212896 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.213117 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.213008 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.213281 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.213261 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704ab7da-672d-4ed0-8efa-7177dde795b0-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.215042 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.215015 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.215231 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.215212 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.255208 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.255163 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" event={"ID":"cf8701ec-59c6-45e1-a62f-f50d102f9758","Type":"ContainerStarted","Data":"dd10718f2efadf3cdacba77c8a8375023dfdfe6dc7d92ea70052f4f01d6faf46"} Oct 31 19:37:45.255378 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.255235 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" event={"ID":"cf8701ec-59c6-45e1-a62f-f50d102f9758","Type":"ContainerStarted","Data":"f5bc2f55a18f73f9b6d05ef6aa1de69c544d589bfd0dd7b4b5f9ffe9ce29a5b3"} Oct 31 19:37:45.256919 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.256893 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" event={"ID":"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68","Type":"ContainerStarted","Data":"6b5cfb6793e485acdd53250e965f475f85045545ab2291f3ddb26d367a169bbb"} Oct 31 19:37:45.257025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.256930 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" event={"ID":"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68","Type":"ContainerStarted","Data":"75f985a13dbc2318a59422de867e72919b697fbef4d193fcd28555b88b9b8254"} Oct 31 19:37:45.266207 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.266183 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9mg7\" (UniqueName: \"kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-kube-api-access-t9mg7\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.266438 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.266418 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.329809 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.329771 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:45.367473 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.367362 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" podStartSLOduration=2.367340853 podStartE2EDuration="2.367340853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:45.367083646 +0000 UTC m=+1237.000477554" watchObservedRunningTime="2026-10-31 19:37:45.367340853 +0000 UTC m=+1237.000734762" Oct 31 19:37:45.410297 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.410261 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:45.416812 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.416622 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:45.477221 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.477162 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" podStartSLOduration=1.477141295 podStartE2EDuration="1.477141295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:45.475510378 +0000 UTC m=+1237.108904288" watchObservedRunningTime="2026-10-31 19:37:45.477141295 +0000 UTC m=+1237.110535203" Oct 31 19:37:45.708384 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.708360 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl"] Oct 31 19:37:45.710486 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:37:45.710459 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704ab7da_672d_4ed0_8efa_7177dde795b0.slice/crio-789f91bea7cd886825db00bdd8b549f8dab6c27a8b727740093d8718acd55085 WatchSource:0}: Error finding container 789f91bea7cd886825db00bdd8b549f8dab6c27a8b727740093d8718acd55085: Status 404 returned error can't find the container with id 789f91bea7cd886825db00bdd8b549f8dab6c27a8b727740093d8718acd55085 Oct 31 19:37:45.713044 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.713016 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:37:45.713132 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.713086 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:37:45.713132 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:45.713114 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:37:46.100079 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:46.100043 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:46.105560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:46.105534 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:46.263997 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:46.263956 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" event={"ID":"704ab7da-672d-4ed0-8efa-7177dde795b0","Type":"ContainerStarted","Data":"0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059"} Oct 31 19:37:46.264167 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:46.264021 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" event={"ID":"704ab7da-672d-4ed0-8efa-7177dde795b0","Type":"ContainerStarted","Data":"789f91bea7cd886825db00bdd8b549f8dab6c27a8b727740093d8718acd55085"} Oct 31 19:37:46.264227 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:46.264216 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:46.264464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:46.264418 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:46.265215 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:46.265198 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:37:46.265311 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:46.265272 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:37:46.330898 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:46.330853 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:46.336107 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:46.336079 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:46.375701 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:46.375593 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" podStartSLOduration=2.375579903 podStartE2EDuration="2.375579903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:46.373225457 +0000 UTC m=+1238.006619365" watchObservedRunningTime="2026-10-31 19:37:46.375579903 +0000 UTC m=+1238.008973810" Oct 31 19:37:47.270369 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:47.270332 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:47.271152 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:47.271132 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:37:50.403207 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:37:50.403171 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:37:53.949499 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:53.949458 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5"] Oct 31 19:37:53.953275 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:53.953252 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" Oct 31 19:37:54.001168 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.001124 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4wgn\" (UniqueName: \"kubernetes.io/projected/b5f42617-d805-4421-9110-f23649003317-kube-api-access-d4wgn\") pod \"payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5\" (UID: \"b5f42617-d805-4421-9110-f23649003317\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" Oct 31 19:37:54.001168 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.001173 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f42617-d805-4421-9110-f23649003317-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5\" (UID: \"b5f42617-d805-4421-9110-f23649003317\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" Oct 31 19:37:54.008131 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.008097 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5"] Oct 31 19:37:54.051004 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.050964 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-5a4438\"" Oct 31 19:37:54.051184 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.051071 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-5a4438-dockercfg-9j4w2\"" Oct 31 19:37:54.102553 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.102518 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wgn\" (UniqueName: \"kubernetes.io/projected/b5f42617-d805-4421-9110-f23649003317-kube-api-access-d4wgn\") pod \"payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5\" (UID: \"b5f42617-d805-4421-9110-f23649003317\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" Oct 31 19:37:54.102553 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.102560 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f42617-d805-4421-9110-f23649003317-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5\" (UID: \"b5f42617-d805-4421-9110-f23649003317\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" Oct 31 19:37:54.103314 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.103295 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f42617-d805-4421-9110-f23649003317-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5\" (UID: \"b5f42617-d805-4421-9110-f23649003317\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" Oct 31 19:37:54.129437 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.129396 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4"] Oct 31 19:37:54.133263 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.133240 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" Oct 31 19:37:54.156048 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.156022 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4wgn\" (UniqueName: \"kubernetes.io/projected/b5f42617-d805-4421-9110-f23649003317-kube-api-access-d4wgn\") pod \"payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5\" (UID: \"b5f42617-d805-4421-9110-f23649003317\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" Oct 31 19:37:54.204231 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.204144 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4\" (UID: \"d12965bb-f23f-41cf-b6cc-592e86d1d8f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" Oct 31 19:37:54.204231 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.204198 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8d8d\" (UniqueName: \"kubernetes.io/projected/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-kube-api-access-v8d8d\") pod \"payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4\" (UID: \"d12965bb-f23f-41cf-b6cc-592e86d1d8f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" Oct 31 19:37:54.207670 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.207635 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4"] Oct 31 19:37:54.267012 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.266974 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" Oct 31 19:37:54.305235 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.305206 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4\" (UID: \"d12965bb-f23f-41cf-b6cc-592e86d1d8f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" Oct 31 19:37:54.305420 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.305263 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8d8d\" (UniqueName: \"kubernetes.io/projected/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-kube-api-access-v8d8d\") pod \"payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4\" (UID: \"d12965bb-f23f-41cf-b6cc-592e86d1d8f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" Oct 31 19:37:54.305982 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.305963 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4\" (UID: \"d12965bb-f23f-41cf-b6cc-592e86d1d8f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" Oct 31 19:37:54.362500 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.362458 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8d8d\" (UniqueName: \"kubernetes.io/projected/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-kube-api-access-v8d8d\") pod \"payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4\" (UID: \"d12965bb-f23f-41cf-b6cc-592e86d1d8f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" Oct 31 19:37:54.446325 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.446286 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" Oct 31 19:37:54.636514 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.636485 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5"] Oct 31 19:37:54.638532 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:37:54.638504 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f42617_d805_4421_9110_f23649003317.slice/crio-82732f7dbc1f732048df7175151cf72356673f229a00c13eeda34561a81d1d0c WatchSource:0}: Error finding container 82732f7dbc1f732048df7175151cf72356673f229a00c13eeda34561a81d1d0c: Status 404 returned error can't find the container with id 82732f7dbc1f732048df7175151cf72356673f229a00c13eeda34561a81d1d0c Oct 31 19:37:54.812756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:54.812727 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4"] Oct 31 19:37:54.814994 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:37:54.814960 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12965bb_f23f_41cf_b6cc_592e86d1d8f7.slice/crio-e69aaadddd0befc71e240a647904adf5d8b15410c593477e6157c01a81e6eb87 WatchSource:0}: Error finding container e69aaadddd0befc71e240a647904adf5d8b15410c593477e6157c01a81e6eb87: Status 404 returned error can't find the container with id e69aaadddd0befc71e240a647904adf5d8b15410c593477e6157c01a81e6eb87 Oct 31 19:37:55.313533 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:55.313493 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" event={"ID":"b5f42617-d805-4421-9110-f23649003317","Type":"ContainerStarted","Data":"82732f7dbc1f732048df7175151cf72356673f229a00c13eeda34561a81d1d0c"} Oct 31 19:37:55.314728 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:55.314703 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" event={"ID":"d12965bb-f23f-41cf-b6cc-592e86d1d8f7","Type":"ContainerStarted","Data":"e69aaadddd0befc71e240a647904adf5d8b15410c593477e6157c01a81e6eb87"} Oct 31 19:37:58.341750 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:58.341706 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" event={"ID":"d12965bb-f23f-41cf-b6cc-592e86d1d8f7","Type":"ContainerStarted","Data":"280664fd9a82876e8b64e7f9fa62555c38f59f4e0010646877af36779ba4df1f"} Oct 31 19:37:58.342237 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:58.341868 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" Oct 31 19:37:58.343406 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:58.343376 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" event={"ID":"b5f42617-d805-4421-9110-f23649003317","Type":"ContainerStarted","Data":"530400e396256edf95b9ee1a803f1f38100883a5135f918d97fe0e7d730607f2"} Oct 31 19:37:58.343544 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:58.343530 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" Oct 31 19:37:58.452485 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:58.452423 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" podStartSLOduration=1.564435319 podStartE2EDuration="4.452404514s" totalImagesPullingTime="2.887969195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:58.451297287 +0000 UTC m=+1250.084691195" watchObservedRunningTime="2026-10-31 19:37:58.452404514 +0000 UTC m=+1250.085798424" Oct 31 19:37:58.558936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:58.558874 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" podStartSLOduration=2.696950008 podStartE2EDuration="5.558859402s" totalImagesPullingTime="2.861909394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:58.556580283 +0000 UTC m=+1250.189974191" watchObservedRunningTime="2026-10-31 19:37:58.558859402 +0000 UTC m=+1250.192253309" Oct 31 19:37:59.072179 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.072102 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl"] Oct 31 19:37:59.075780 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.075760 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" Oct 31 19:37:59.123937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.123876 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b5155c\"" Oct 31 19:37:59.123937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.123876 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b5155c-dockercfg-pnxnm\"" Oct 31 19:37:59.158930 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.158869 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl\" (UID: \"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" Oct 31 19:37:59.159080 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.159047 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vffr4\" (UniqueName: \"kubernetes.io/projected/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-kube-api-access-vffr4\") pod \"payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl\" (UID: \"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" Oct 31 19:37:59.171757 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.171724 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl"] Oct 31 19:37:59.260105 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.260073 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl\" (UID: \"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" Oct 31 19:37:59.260284 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.260218 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vffr4\" (UniqueName: \"kubernetes.io/projected/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-kube-api-access-vffr4\") pod \"payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl\" (UID: \"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" Oct 31 19:37:59.260807 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.260787 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl\" (UID: \"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" Oct 31 19:37:59.313282 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.313255 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vffr4\" (UniqueName: \"kubernetes.io/projected/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-kube-api-access-vffr4\") pod \"payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl\" (UID: \"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" Oct 31 19:37:59.388839 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.388761 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" Oct 31 19:37:59.751160 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:37:59.751073 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl"] Oct 31 19:37:59.752482 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:37:59.752448 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5ade3d_0e5f_429d_a49c_2c78ed7c1f35.slice/crio-8cdc12eac6e5f56304daab47c6eb7121b7040647d5b37ff17f1357df447a49ac WatchSource:0}: Error finding container 8cdc12eac6e5f56304daab47c6eb7121b7040647d5b37ff17f1357df447a49ac: Status 404 returned error can't find the container with id 8cdc12eac6e5f56304daab47c6eb7121b7040647d5b37ff17f1357df447a49ac Oct 31 19:38:00.354319 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:00.354287 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" event={"ID":"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35","Type":"ContainerStarted","Data":"7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f"} Oct 31 19:38:00.354487 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:00.354341 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" event={"ID":"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35","Type":"ContainerStarted","Data":"8cdc12eac6e5f56304daab47c6eb7121b7040647d5b37ff17f1357df447a49ac"} Oct 31 19:38:00.354487 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:00.354441 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" Oct 31 19:38:00.459686 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:00.459636 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" podStartSLOduration=1.459621909 podStartE2EDuration="1.459621909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:00.459134534 +0000 UTC m=+1252.092528443" watchObservedRunningTime="2026-10-31 19:38:00.459621909 +0000 UTC m=+1252.093015816" Oct 31 19:38:01.524022 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:01.523987 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl"] Oct 31 19:38:01.527694 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:01.527674 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" Oct 31 19:38:01.585042 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:01.584998 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801671a2-4d7f-4734-8a10-1d7975dda195-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl\" (UID: \"801671a2-4d7f-4734-8a10-1d7975dda195\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" Oct 31 19:38:01.586571 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:01.586539 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7z5r\" (UniqueName: \"kubernetes.io/projected/801671a2-4d7f-4734-8a10-1d7975dda195-kube-api-access-t7z5r\") pod \"payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl\" (UID: \"801671a2-4d7f-4734-8a10-1d7975dda195\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" Oct 31 19:38:01.624938 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:01.624373 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-570915\"" Oct 31 19:38:01.624938 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:01.624458 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-570915-dockercfg-z7ctv\"" Oct 31 19:38:01.626937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:01.626905 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl"] Oct 31 19:38:01.687783 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:01.687749 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7z5r\" (UniqueName: \"kubernetes.io/projected/801671a2-4d7f-4734-8a10-1d7975dda195-kube-api-access-t7z5r\") pod \"payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl\" (UID: \"801671a2-4d7f-4734-8a10-1d7975dda195\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" Oct 31 19:38:01.687991 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:01.687853 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801671a2-4d7f-4734-8a10-1d7975dda195-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl\" (UID: \"801671a2-4d7f-4734-8a10-1d7975dda195\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" Oct 31 19:38:01.688526 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:01.688506 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801671a2-4d7f-4734-8a10-1d7975dda195-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl\" (UID: \"801671a2-4d7f-4734-8a10-1d7975dda195\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" Oct 31 19:38:01.742427 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:01.742394 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7z5r\" (UniqueName: \"kubernetes.io/projected/801671a2-4d7f-4734-8a10-1d7975dda195-kube-api-access-t7z5r\") pod \"payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl\" (UID: \"801671a2-4d7f-4734-8a10-1d7975dda195\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" Oct 31 19:38:01.840550 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:01.840459 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" Oct 31 19:38:02.205949 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:02.205923 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl"] Oct 31 19:38:02.208104 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:38:02.208075 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801671a2_4d7f_4734_8a10_1d7975dda195.slice/crio-6c5c338340a8d8d87dd753e63115d33b6510baddeeb25fcb86d008d1c6b6b49b WatchSource:0}: Error finding container 6c5c338340a8d8d87dd753e63115d33b6510baddeeb25fcb86d008d1c6b6b49b: Status 404 returned error can't find the container with id 6c5c338340a8d8d87dd753e63115d33b6510baddeeb25fcb86d008d1c6b6b49b Oct 31 19:38:02.367252 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:02.367218 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" event={"ID":"801671a2-4d7f-4734-8a10-1d7975dda195","Type":"ContainerStarted","Data":"cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80"} Oct 31 19:38:02.367445 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:02.367276 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" event={"ID":"801671a2-4d7f-4734-8a10-1d7975dda195","Type":"ContainerStarted","Data":"6c5c338340a8d8d87dd753e63115d33b6510baddeeb25fcb86d008d1c6b6b49b"} Oct 31 19:38:02.367507 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:02.367438 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" Oct 31 19:38:02.474051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:02.473835 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" podStartSLOduration=1.4738161060000001 podStartE2EDuration="1.473816106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:02.473258523 +0000 UTC m=+1254.106652450" watchObservedRunningTime="2026-10-31 19:38:02.473816106 +0000 UTC m=+1254.107210015" Oct 31 19:38:04.217132 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.217094 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr"] Oct 31 19:38:04.221138 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.221115 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" Oct 31 19:38:04.274727 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.274698 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr"] Oct 31 19:38:04.317775 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.317737 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-7cfafd\"" Oct 31 19:38:04.317775 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.317743 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-7cfafd-dockercfg-cw5cz\"" Oct 31 19:38:04.318080 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.318058 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr\" (UID: \"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" Oct 31 19:38:04.318140 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.318081 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xzrr\" (UniqueName: \"kubernetes.io/projected/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-kube-api-access-5xzrr\") pod \"payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr\" (UID: \"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" Oct 31 19:38:04.419252 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.419209 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr\" (UID: \"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" Oct 31 19:38:04.419252 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.419254 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xzrr\" (UniqueName: \"kubernetes.io/projected/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-kube-api-access-5xzrr\") pod \"payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr\" (UID: \"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" Oct 31 19:38:04.419953 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.419935 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr\" (UID: \"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" Oct 31 19:38:04.473006 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.472958 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xzrr\" (UniqueName: \"kubernetes.io/projected/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-kube-api-access-5xzrr\") pod \"payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr\" (UID: \"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" Oct 31 19:38:04.533990 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.533960 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" Oct 31 19:38:04.900699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:04.900508 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr"] Oct 31 19:38:04.903757 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:38:04.903729 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37adad1_4605_4dfc_b01c_7b9c6b9bb2d8.slice/crio-2577d6a9efbf4ddfe235fb8d467cfe33547feb707e3ff4851793cbc7b89a3e70 WatchSource:0}: Error finding container 2577d6a9efbf4ddfe235fb8d467cfe33547feb707e3ff4851793cbc7b89a3e70: Status 404 returned error can't find the container with id 2577d6a9efbf4ddfe235fb8d467cfe33547feb707e3ff4851793cbc7b89a3e70 Oct 31 19:38:05.389952 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:05.389912 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" event={"ID":"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8","Type":"ContainerStarted","Data":"ef3b627586ea08bbf9f88fa20358d1b5a10bbc42133c3f5c964637b9abf6abad"} Oct 31 19:38:05.390377 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:05.390015 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" Oct 31 19:38:05.390377 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:05.390041 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" event={"ID":"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8","Type":"ContainerStarted","Data":"2577d6a9efbf4ddfe235fb8d467cfe33547feb707e3ff4851793cbc7b89a3e70"} Oct 31 19:38:05.542372 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:05.542314 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" podStartSLOduration=1.542299249 podStartE2EDuration="1.542299249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:05.49514766 +0000 UTC m=+1257.128541574" watchObservedRunningTime="2026-10-31 19:38:05.542299249 +0000 UTC m=+1257.175693156" Oct 31 19:38:09.351966 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:09.351935 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" Oct 31 19:38:09.352377 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:09.352023 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" Oct 31 19:38:10.826611 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.826560 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd"] Oct 31 19:38:10.832140 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.832117 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:10.887346 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.887316 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-maas-api-tls\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:10.887510 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.887464 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-metrics-certs\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:10.887564 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.887506 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:10.887564 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.887535 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4zzv\" (UniqueName: \"kubernetes.io/projected/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-kube-api-access-q4zzv\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:10.942907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.933187 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-093b97-serving-cert\"" Oct 31 19:38:10.949048 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.949017 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd"] Oct 31 19:38:10.988709 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.988657 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-maas-api-tls\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:10.988924 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.988783 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-metrics-certs\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:10.988924 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.988825 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:10.988924 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.988845 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4zzv\" (UniqueName: \"kubernetes.io/projected/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-kube-api-access-q4zzv\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:10.988924 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:38:10.988848 6107 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-serving-cert: secret "maas-api-e2e-worker-w0-093b97-serving-cert" not found Oct 31 19:38:10.989157 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:38:10.988953 6107 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-maas-api-tls podName:4831dbcb-ea5f-4862-8250-cbe24c6fbd81 nodeName:}" failed. No retries permitted until 2026-10-31 19:38:11.488927608 +0000 UTC m=+1263.122321516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-maas-api-tls") pod "maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" (UID: "4831dbcb-ea5f-4862-8250-cbe24c6fbd81") : secret "maas-api-e2e-worker-w0-093b97-serving-cert" not found Oct 31 19:38:10.989532 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.989511 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:10.991412 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:10.991387 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-metrics-certs\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:11.042293 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:11.042265 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4zzv\" (UniqueName: \"kubernetes.io/projected/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-kube-api-access-q4zzv\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:11.363341 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:11.363317 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" Oct 31 19:38:11.494640 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:11.494604 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-maas-api-tls\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:11.497271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:11.497242 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-maas-api-tls\") pod \"maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:11.746680 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:11.746642 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:12.129064 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:12.129030 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd"] Oct 31 19:38:12.131931 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:38:12.131868 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4831dbcb_ea5f_4862_8250_cbe24c6fbd81.slice/crio-fecfbbff1690bff51bd9a3b5c04e95f2c7a547d78c3f1a1a8e9ef9da01fc2388 WatchSource:0}: Error finding container fecfbbff1690bff51bd9a3b5c04e95f2c7a547d78c3f1a1a8e9ef9da01fc2388: Status 404 returned error can't find the container with id fecfbbff1690bff51bd9a3b5c04e95f2c7a547d78c3f1a1a8e9ef9da01fc2388 Oct 31 19:38:12.452135 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:12.452050 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" event={"ID":"4831dbcb-ea5f-4862-8250-cbe24c6fbd81","Type":"ContainerStarted","Data":"fecfbbff1690bff51bd9a3b5c04e95f2c7a547d78c3f1a1a8e9ef9da01fc2388"} Oct 31 19:38:13.375227 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:13.375147 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" Oct 31 19:38:13.459609 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:13.459569 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" event={"ID":"4831dbcb-ea5f-4862-8250-cbe24c6fbd81","Type":"ContainerStarted","Data":"f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3"} Oct 31 19:38:13.459770 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:13.459642 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:13.596467 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:13.596414 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" podStartSLOduration=2.734519524 podStartE2EDuration="3.596398049s" totalImagesPullingTime="861.878525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:13.594485925 +0000 UTC m=+1265.227879833" watchObservedRunningTime="2026-10-31 19:38:13.596398049 +0000 UTC m=+1265.229791956" Oct 31 19:38:16.136852 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.136812 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr"] Oct 31 19:38:16.141150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.141124 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" Oct 31 19:38:16.161216 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.161186 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr"] Oct 31 19:38:16.189743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.189712 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-093b97\"" Oct 31 19:38:16.189950 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.189870 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-093b97-dockercfg-995cd\"" Oct 31 19:38:16.239596 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.239561 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267"] Oct 31 19:38:16.244922 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.244351 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" Oct 31 19:38:16.279558 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.279529 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de419c01-12e0-4606-a362-666c83e6c7ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr\" (UID: \"de419c01-12e0-4606-a362-666c83e6c7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" Oct 31 19:38:16.279738 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.279581 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk42v\" (UniqueName: \"kubernetes.io/projected/de419c01-12e0-4606-a362-666c83e6c7ee-kube-api-access-qk42v\") pod \"payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr\" (UID: \"de419c01-12e0-4606-a362-666c83e6c7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" Oct 31 19:38:16.296616 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.296582 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267"] Oct 31 19:38:16.380364 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.380326 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de419c01-12e0-4606-a362-666c83e6c7ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr\" (UID: \"de419c01-12e0-4606-a362-666c83e6c7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" Oct 31 19:38:16.380533 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.380429 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk42v\" (UniqueName: \"kubernetes.io/projected/de419c01-12e0-4606-a362-666c83e6c7ee-kube-api-access-qk42v\") pod \"payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr\" (UID: \"de419c01-12e0-4606-a362-666c83e6c7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" Oct 31 19:38:16.380533 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.380485 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66dba4ff-acee-4801-9e65-dd0198d23537-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267\" (UID: \"66dba4ff-acee-4801-9e65-dd0198d23537\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" Oct 31 19:38:16.380636 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.380549 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7x6m\" (UniqueName: \"kubernetes.io/projected/66dba4ff-acee-4801-9e65-dd0198d23537-kube-api-access-n7x6m\") pod \"payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267\" (UID: \"66dba4ff-acee-4801-9e65-dd0198d23537\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" Oct 31 19:38:16.381138 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.381120 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de419c01-12e0-4606-a362-666c83e6c7ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr\" (UID: \"de419c01-12e0-4606-a362-666c83e6c7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" Oct 31 19:38:16.398252 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.398195 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" Oct 31 19:38:16.433300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.433265 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk42v\" (UniqueName: \"kubernetes.io/projected/de419c01-12e0-4606-a362-666c83e6c7ee-kube-api-access-qk42v\") pod \"payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr\" (UID: \"de419c01-12e0-4606-a362-666c83e6c7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" Oct 31 19:38:16.462273 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.462240 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" Oct 31 19:38:16.481345 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.481319 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7x6m\" (UniqueName: \"kubernetes.io/projected/66dba4ff-acee-4801-9e65-dd0198d23537-kube-api-access-n7x6m\") pod \"payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267\" (UID: \"66dba4ff-acee-4801-9e65-dd0198d23537\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" Oct 31 19:38:16.481500 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.481478 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66dba4ff-acee-4801-9e65-dd0198d23537-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267\" (UID: \"66dba4ff-acee-4801-9e65-dd0198d23537\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" Oct 31 19:38:16.482429 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.482409 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66dba4ff-acee-4801-9e65-dd0198d23537-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267\" (UID: \"66dba4ff-acee-4801-9e65-dd0198d23537\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" Oct 31 19:38:16.543962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.543864 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7x6m\" (UniqueName: \"kubernetes.io/projected/66dba4ff-acee-4801-9e65-dd0198d23537-kube-api-access-n7x6m\") pod \"payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267\" (UID: \"66dba4ff-acee-4801-9e65-dd0198d23537\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" Oct 31 19:38:16.560006 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.559700 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" Oct 31 19:38:16.835909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.835862 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr"] Oct 31 19:38:16.837742 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:38:16.837700 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde419c01_12e0_4606_a362_666c83e6c7ee.slice/crio-5e623b328e078a5c2f640969bcf1c3dad8a12d7cb3a7fcf1510e4bbea6945706 WatchSource:0}: Error finding container 5e623b328e078a5c2f640969bcf1c3dad8a12d7cb3a7fcf1510e4bbea6945706: Status 404 returned error can't find the container with id 5e623b328e078a5c2f640969bcf1c3dad8a12d7cb3a7fcf1510e4bbea6945706 Oct 31 19:38:16.941934 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:16.941905 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267"] Oct 31 19:38:16.943946 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:38:16.943910 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66dba4ff_acee_4801_9e65_dd0198d23537.slice/crio-1331dd00ca0b4f06713512535a49048006ed1e52c09cdeed216150030d611bbe WatchSource:0}: Error finding container 1331dd00ca0b4f06713512535a49048006ed1e52c09cdeed216150030d611bbe: Status 404 returned error can't find the container with id 1331dd00ca0b4f06713512535a49048006ed1e52c09cdeed216150030d611bbe Oct 31 19:38:17.486453 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:17.486420 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" event={"ID":"de419c01-12e0-4606-a362-666c83e6c7ee","Type":"ContainerStarted","Data":"2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c"} Oct 31 19:38:17.488060 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:17.486480 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" event={"ID":"de419c01-12e0-4606-a362-666c83e6c7ee","Type":"ContainerStarted","Data":"5e623b328e078a5c2f640969bcf1c3dad8a12d7cb3a7fcf1510e4bbea6945706"} Oct 31 19:38:17.488060 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:17.486676 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" Oct 31 19:38:17.488347 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:17.488324 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" event={"ID":"66dba4ff-acee-4801-9e65-dd0198d23537","Type":"ContainerStarted","Data":"81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194"} Oct 31 19:38:17.488477 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:17.488364 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" event={"ID":"66dba4ff-acee-4801-9e65-dd0198d23537","Type":"ContainerStarted","Data":"1331dd00ca0b4f06713512535a49048006ed1e52c09cdeed216150030d611bbe"} Oct 31 19:38:17.488477 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:17.488442 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" Oct 31 19:38:17.536078 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:17.535864 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:42782->10.133.0.42:15021: read: connection reset by peer" start-of-body= Oct 31 19:38:17.536078 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:17.535948 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:42782->10.133.0.42:15021: read: connection reset by peer" Oct 31 19:38:17.596594 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:17.596533 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" podStartSLOduration=1.5965099600000001 podStartE2EDuration="1.59650996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:17.590952303 +0000 UTC m=+1269.224346212" watchObservedRunningTime="2026-10-31 19:38:17.59650996 +0000 UTC m=+1269.229903868" Oct 31 19:38:17.720012 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:17.719958 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" podStartSLOduration=1.719945734 podStartE2EDuration="1.719945734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:17.719787198 +0000 UTC m=+1269.353181098" watchObservedRunningTime="2026-10-31 19:38:17.719945734 +0000 UTC m=+1269.353339642" Oct 31 19:38:18.494794 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.494708 6107 generic.go:356] "Generic (PLEG): container finished" podID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerID="0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059" exitCode=0 Oct 31 19:38:18.495199 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.494789 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" event={"ID":"704ab7da-672d-4ed0-8efa-7177dde795b0","Type":"ContainerDied","Data":"0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059"} Oct 31 19:38:18.495384 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.495354 6107 scope.go:122] "RemoveContainer" containerID="0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059" Oct 31 19:38:18.496517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.496495 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:38:18.496572 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.496560 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:38:18.496618 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.496591 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:38:18.781744 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.781656 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq"] Oct 31 19:38:18.784999 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.784957 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" Oct 31 19:38:18.821622 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.821588 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swfh\" (UniqueName: \"kubernetes.io/projected/b05e3c93-830d-40fa-b368-1b8f23413bd8-kube-api-access-9swfh\") pod \"payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq\" (UID: \"b05e3c93-830d-40fa-b368-1b8f23413bd8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" Oct 31 19:38:18.821794 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.821682 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05e3c93-830d-40fa-b368-1b8f23413bd8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq\" (UID: \"b05e3c93-830d-40fa-b368-1b8f23413bd8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" Oct 31 19:38:18.833397 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.833371 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-967dee-dockercfg-qxcx5\"" Oct 31 19:38:18.836468 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.836439 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-967dee\"" Oct 31 19:38:18.840240 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.840212 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq"] Oct 31 19:38:18.922273 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.922241 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swfh\" (UniqueName: \"kubernetes.io/projected/b05e3c93-830d-40fa-b368-1b8f23413bd8-kube-api-access-9swfh\") pod \"payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq\" (UID: \"b05e3c93-830d-40fa-b368-1b8f23413bd8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" Oct 31 19:38:18.922503 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.922295 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05e3c93-830d-40fa-b368-1b8f23413bd8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq\" (UID: \"b05e3c93-830d-40fa-b368-1b8f23413bd8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" Oct 31 19:38:18.923091 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.923071 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05e3c93-830d-40fa-b368-1b8f23413bd8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq\" (UID: \"b05e3c93-830d-40fa-b368-1b8f23413bd8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" Oct 31 19:38:18.976100 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:18.976073 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swfh\" (UniqueName: \"kubernetes.io/projected/b05e3c93-830d-40fa-b368-1b8f23413bd8-kube-api-access-9swfh\") pod \"payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq\" (UID: \"b05e3c93-830d-40fa-b368-1b8f23413bd8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" Oct 31 19:38:19.098701 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:19.098627 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" Oct 31 19:38:19.473533 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:19.473511 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 19:38:19.492053 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:19.492024 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq"] Oct 31 19:38:19.493793 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:38:19.493764 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05e3c93_830d_40fa_b368_1b8f23413bd8.slice/crio-a5895634d93844bc97562128b13cabf3511dd77ff8c2251a4fc7625cbf9d147b WatchSource:0}: Error finding container a5895634d93844bc97562128b13cabf3511dd77ff8c2251a4fc7625cbf9d147b: Status 404 returned error can't find the container with id a5895634d93844bc97562128b13cabf3511dd77ff8c2251a4fc7625cbf9d147b Oct 31 19:38:19.528466 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:19.528429 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" event={"ID":"704ab7da-672d-4ed0-8efa-7177dde795b0","Type":"ContainerStarted","Data":"792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5"} Oct 31 19:38:19.529966 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:19.529910 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" event={"ID":"b05e3c93-830d-40fa-b368-1b8f23413bd8","Type":"ContainerStarted","Data":"a5895634d93844bc97562128b13cabf3511dd77ff8c2251a4fc7625cbf9d147b"} Oct 31 19:38:20.330943 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:20.330905 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:38:20.331159 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:20.331134 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 31 19:38:20.331204 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:20.331188 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 31 19:38:20.429903 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:38:20.429488 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:38:20.536637 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:20.536604 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" event={"ID":"b05e3c93-830d-40fa-b368-1b8f23413bd8","Type":"ContainerStarted","Data":"2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753"} Oct 31 19:38:20.537050 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:20.536702 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" Oct 31 19:38:20.642127 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:20.642027 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" podStartSLOduration=2.642009127 podStartE2EDuration="2.642009127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:20.64163286 +0000 UTC m=+1272.275026771" watchObservedRunningTime="2026-10-31 19:38:20.642009127 +0000 UTC m=+1272.275403035" Oct 31 19:38:21.331387 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:21.331350 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 31 19:38:21.331608 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:21.331413 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 31 19:38:22.331065 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:22.331027 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 31 19:38:22.331546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:22.331091 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 31 19:38:23.331268 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:23.331233 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 31 19:38:23.331786 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:23.331290 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 31 19:38:24.330720 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:24.330681 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 31 19:38:24.330933 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:24.330740 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 31 19:38:25.334611 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:25.334580 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:38:25.334997 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:25.334846 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:38:25.335518 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:25.335503 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 19:38:28.498608 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:28.498568 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" Oct 31 19:38:28.499054 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:28.498635 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" Oct 31 19:38:31.306830 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.306795 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9"] Oct 31 19:38:31.312976 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.312952 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.385528 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.385487 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.385726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.385559 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-metrics-certs\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.385784 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.385731 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7c440a-7e60-40a8-a55b-27d8760c990e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.385924 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.385899 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwbh\" (UniqueName: \"kubernetes.io/projected/8e7c440a-7e60-40a8-a55b-27d8760c990e-kube-api-access-smwbh\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.409616 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.409576 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-7cfafd-serving-cert\"" Oct 31 19:38:31.412155 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.412128 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9"] Oct 31 19:38:31.486857 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.486821 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.487061 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.486923 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-metrics-certs\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.487061 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.487001 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7c440a-7e60-40a8-a55b-27d8760c990e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.487061 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:38:31.487013 6107 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-serving-cert: secret "maas-api-e2e-worker-w4-7cfafd-serving-cert" not found Oct 31 19:38:31.487227 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:38:31.487101 6107 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-maas-api-tls podName:8e7c440a-7e60-40a8-a55b-27d8760c990e nodeName:}" failed. No retries permitted until 2026-10-31 19:38:31.987077282 +0000 UTC m=+1283.620471171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-maas-api-tls") pod "maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" (UID: "8e7c440a-7e60-40a8-a55b-27d8760c990e") : secret "maas-api-e2e-worker-w4-7cfafd-serving-cert" not found Oct 31 19:38:31.487227 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.487127 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwbh\" (UniqueName: \"kubernetes.io/projected/8e7c440a-7e60-40a8-a55b-27d8760c990e-kube-api-access-smwbh\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.488126 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.488088 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7c440a-7e60-40a8-a55b-27d8760c990e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.490023 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.490001 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-metrics-certs\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.544546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.544507 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwbh\" (UniqueName: \"kubernetes.io/projected/8e7c440a-7e60-40a8-a55b-27d8760c990e-kube-api-access-smwbh\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.545354 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.545328 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" Oct 31 19:38:31.993203 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.993169 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:31.995690 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:31.995656 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:32.226759 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:32.226718 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:32.308109 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:32.308067 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:38:32.308524 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:32.308182 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:32.604227 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:32.604186 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9"] Oct 31 19:38:32.605372 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:38:32.605346 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7c440a_7e60_40a8_a55b_27d8760c990e.slice/crio-678ed2fe1d501b1bf76425695e0daed8ec2f033258b772440a0d53d6f59081ad WatchSource:0}: Error finding container 678ed2fe1d501b1bf76425695e0daed8ec2f033258b772440a0d53d6f59081ad: Status 404 returned error can't find the container with id 678ed2fe1d501b1bf76425695e0daed8ec2f033258b772440a0d53d6f59081ad Oct 31 19:38:32.643142 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:32.643103 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" event={"ID":"8e7c440a-7e60-40a8-a55b-27d8760c990e","Type":"ContainerStarted","Data":"678ed2fe1d501b1bf76425695e0daed8ec2f033258b772440a0d53d6f59081ad"} Oct 31 19:38:33.650942 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:33.650895 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" event={"ID":"8e7c440a-7e60-40a8-a55b-27d8760c990e","Type":"ContainerStarted","Data":"3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c"} Oct 31 19:38:33.651324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:33.651032 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:33.758460 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:33.758411 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" podStartSLOduration=1.921134458 podStartE2EDuration="2.758396431s" totalImagesPullingTime="837.261973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:33.756300133 +0000 UTC m=+1285.389694184" watchObservedRunningTime="2026-10-31 19:38:33.758396431 +0000 UTC m=+1285.391790338" Oct 31 19:38:34.659357 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:34.659323 6107 generic.go:356] "Generic (PLEG): container finished" podID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerID="dd10718f2efadf3cdacba77c8a8375023dfdfe6dc7d92ea70052f4f01d6faf46" exitCode=0 Oct 31 19:38:34.659904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:34.659431 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" event={"ID":"cf8701ec-59c6-45e1-a62f-f50d102f9758","Type":"ContainerDied","Data":"dd10718f2efadf3cdacba77c8a8375023dfdfe6dc7d92ea70052f4f01d6faf46"} Oct 31 19:38:34.660146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:34.660127 6107 scope.go:122] "RemoveContainer" containerID="dd10718f2efadf3cdacba77c8a8375023dfdfe6dc7d92ea70052f4f01d6faf46" Oct 31 19:38:34.661424 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:34.661400 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:38:34.661488 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:34.661461 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:38:34.661527 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:34.661494 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:38:35.676840 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:35.676802 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" event={"ID":"cf8701ec-59c6-45e1-a62f-f50d102f9758","Type":"ContainerStarted","Data":"eabf1d7586ea8c283d953e91aa4f9b442f191bdf5997872eb1ed6a0426d6c340"} Oct 31 19:38:36.106906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:36.104289 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:38:36.106906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:36.104451 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 31 19:38:36.106906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:36.104494 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 31 19:38:37.105905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:37.104284 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 31 19:38:37.105905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:37.104345 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 31 19:38:38.110905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:38.103992 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 31 19:38:38.110905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:38.104057 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 31 19:38:39.108907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:39.104206 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 31 19:38:39.108907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:39.104268 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 31 19:38:39.665295 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:39.665264 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 19:38:40.116911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:40.107989 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 31 19:38:40.116911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:40.108050 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 31 19:38:41.103844 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:41.103808 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:38:41.104374 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:41.104122 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:38:41.104748 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:41.104727 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 19:38:44.760272 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:44.760237 6107 generic.go:356] "Generic (PLEG): container finished" podID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerID="6b5cfb6793e485acdd53250e965f475f85045545ab2291f3ddb26d367a169bbb" exitCode=0 Oct 31 19:38:44.760969 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:44.760304 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" event={"ID":"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68","Type":"ContainerDied","Data":"6b5cfb6793e485acdd53250e965f475f85045545ab2291f3ddb26d367a169bbb"} Oct 31 19:38:44.760969 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:44.760671 6107 scope.go:122] "RemoveContainer" containerID="6b5cfb6793e485acdd53250e965f475f85045545ab2291f3ddb26d367a169bbb" Oct 31 19:38:44.761938 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:44.761904 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:38:44.762060 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:44.761991 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:38:44.762060 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:44.762035 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:38:45.767353 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:45.767316 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" event={"ID":"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68","Type":"ContainerStarted","Data":"501b362e65d3d431d0d1dabc4d3d819fae7fb9b1bf8cc3892d638f3ef1a3178f"} Oct 31 19:38:46.265039 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:46.265001 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:38:46.411411 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:46.411378 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:38:46.411905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:46.411711 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 31 19:38:46.411905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:46.411756 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 31 19:38:47.415315 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:47.415277 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 31 19:38:47.416052 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:47.416019 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 31 19:38:48.413905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:48.411297 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 31 19:38:48.413905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:48.411357 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 31 19:38:49.412330 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:49.412292 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 31 19:38:49.412911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:49.412858 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 31 19:38:50.413480 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:50.413043 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 31 19:38:50.413480 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:50.413104 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 31 19:38:50.426556 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:38:50.426499 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:38:51.414618 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:51.414588 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:38:51.415613 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:38:51.415594 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:39:20.402811 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:39:20.402770 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:39:50.403505 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:39:50.403423 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:39:51.714379 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.714339 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6"] Oct 31 19:39:51.717156 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.717139 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.767955 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.767922 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"openshift-service-ca.crt\"" Oct 31 19:39:51.813850 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.813820 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"kube-root-ca.crt\"" Oct 31 19:39:51.814061 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.813941 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"default-dockercfg-4cd7k\"" Oct 31 19:39:51.814061 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.814029 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"premium-w3-662be1-kserve-self-signed-certs\"" Oct 31 19:39:51.816414 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.816389 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6"] Oct 31 19:39:51.830388 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.830364 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f638482-f128-45a1-8a74-3d9751414c81-tls-certs\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.830479 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.830407 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-home\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.830529 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.830509 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-tmp-dir\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.830569 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.830547 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-dshm\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.830606 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.830567 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-model-cache\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.830648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.830615 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62pxl\" (UniqueName: \"kubernetes.io/projected/5f638482-f128-45a1-8a74-3d9751414c81-kube-api-access-62pxl\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.932083 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.932043 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f638482-f128-45a1-8a74-3d9751414c81-tls-certs\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.932261 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.932114 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-home\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.932261 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.932171 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-tmp-dir\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.932261 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.932203 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-dshm\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.932261 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.932230 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-model-cache\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.932261 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.932254 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62pxl\" (UniqueName: \"kubernetes.io/projected/5f638482-f128-45a1-8a74-3d9751414c81-kube-api-access-62pxl\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.932615 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.932587 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-home\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.932826 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.932803 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-model-cache\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.932905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.932821 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-tmp-dir\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.934749 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.934728 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f638482-f128-45a1-8a74-3d9751414c81-tls-certs\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.935794 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.935762 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-dshm\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:51.988240 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:51.988205 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62pxl\" (UniqueName: \"kubernetes.io/projected/5f638482-f128-45a1-8a74-3d9751414c81-kube-api-access-62pxl\") pod \"premium-w3-662be1-kserve-5b9b9bc86f-dmrl6\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:52.027689 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:52.027655 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:39:52.395724 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:52.395690 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6"] Oct 31 19:39:52.397590 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:39:52.397552 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f638482_f128_45a1_8a74_3d9751414c81.slice/crio-5e0e2a64ade952cb5dfe338dc3dfe2326730965c95df61f7ad9003f4649ac15c WatchSource:0}: Error finding container 5e0e2a64ade952cb5dfe338dc3dfe2326730965c95df61f7ad9003f4649ac15c: Status 404 returned error can't find the container with id 5e0e2a64ade952cb5dfe338dc3dfe2326730965c95df61f7ad9003f4649ac15c Oct 31 19:39:53.183860 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:53.183823 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" event={"ID":"5f638482-f128-45a1-8a74-3d9751414c81","Type":"ContainerStarted","Data":"0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e"} Oct 31 19:39:53.184326 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:53.183871 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" event={"ID":"5f638482-f128-45a1-8a74-3d9751414c81","Type":"ContainerStarted","Data":"5e0e2a64ade952cb5dfe338dc3dfe2326730965c95df61f7ad9003f4649ac15c"} Oct 31 19:39:53.291162 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:53.291108 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podStartSLOduration=2.291092968 podStartE2EDuration="2.291092968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:53.289450419 +0000 UTC m=+1364.922844332" watchObservedRunningTime="2026-10-31 19:39:53.291092968 +0000 UTC m=+1364.924486875" Oct 31 19:39:54.028233 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:39:54.028203 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:40:02.028867 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:02.028829 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:40:02.041369 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:02.041340 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:40:02.245128 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:02.245097 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:40:08.620475 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.620442 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2"] Oct 31 19:40:08.623650 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.623630 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.709154 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.709123 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tls-certs\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.709322 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.709175 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnvnz\" (UniqueName: \"kubernetes.io/projected/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-kube-api-access-jnvnz\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.709322 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.709298 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tmp-dir\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.709432 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.709401 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-model-cache\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.709484 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.709442 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-home\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.709593 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.709560 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-dshm\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.718461 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.718438 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-967dee\"/\"kube-root-ca.crt\"" Oct 31 19:40:08.718756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.718729 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-967dee\"/\"openshift-service-ca.crt\"" Oct 31 19:40:08.718860 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.718758 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-967dee\"/\"default-dockercfg-s96lf\"" Oct 31 19:40:08.718979 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.718959 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-967dee\"/\"simulator-w2-967dee-kserve-self-signed-certs\"" Oct 31 19:40:08.721396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.721373 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2"] Oct 31 19:40:08.811235 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.811198 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tls-certs\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.811436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.811247 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnvnz\" (UniqueName: \"kubernetes.io/projected/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-kube-api-access-jnvnz\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.811436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.811297 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tmp-dir\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.811436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.811355 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-model-cache\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.811436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.811380 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-home\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.811680 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.811567 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-dshm\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.811804 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.811782 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-home\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.811844 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.811817 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-model-cache\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.811913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.811843 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tmp-dir\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.813619 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.813596 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-dshm\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.814036 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.814019 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tls-certs\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.870839 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.870768 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnvnz\" (UniqueName: \"kubernetes.io/projected/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-kube-api-access-jnvnz\") pod \"simulator-w2-967dee-kserve-588c564d67-bssg2\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:08.934802 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:08.934765 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:09.312629 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:09.312598 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2"] Oct 31 19:40:09.314247 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:40:09.314216 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf9b0180_a73d_4dcb_a8d0_7709b3fc6f04.slice/crio-e330f019c03fffb9f19eeb09be6ba39fce8a4e17eac3e3e6e3001212309de4b1 WatchSource:0}: Error finding container e330f019c03fffb9f19eeb09be6ba39fce8a4e17eac3e3e6e3001212309de4b1: Status 404 returned error can't find the container with id e330f019c03fffb9f19eeb09be6ba39fce8a4e17eac3e3e6e3001212309de4b1 Oct 31 19:40:10.284126 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:10.284082 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" event={"ID":"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04","Type":"ContainerStarted","Data":"6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f"} Oct 31 19:40:10.284544 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:10.284141 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" event={"ID":"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04","Type":"ContainerStarted","Data":"e330f019c03fffb9f19eeb09be6ba39fce8a4e17eac3e3e6e3001212309de4b1"} Oct 31 19:40:10.398368 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:10.398318 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podStartSLOduration=2.398304546 podStartE2EDuration="2.398304546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:10.396799102 +0000 UTC m=+1382.030193010" watchObservedRunningTime="2026-10-31 19:40:10.398304546 +0000 UTC m=+1382.031698453" Oct 31 19:40:10.935445 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:10.935406 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:18.935802 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:18.935769 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:18.948301 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:18.948278 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:19.346311 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:19.346282 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:40:20.403726 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:40:20.403692 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:40:40.259519 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.259479 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg"] Oct 31 19:40:40.262591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.262569 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.360138 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.360094 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-093b97\"/\"kube-root-ca.crt\"" Oct 31 19:40:40.360352 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.360095 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-093b97\"/\"openshift-service-ca.crt\"" Oct 31 19:40:40.360352 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.360157 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-093b97\"/\"default-dockercfg-kds7w\"" Oct 31 19:40:40.360352 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.360218 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-093b97\"/\"premium-w0-093b97-kserve-self-signed-certs\"" Oct 31 19:40:40.362705 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.362684 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg"] Oct 31 19:40:40.364735 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.364711 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-dshm\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.364851 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.364747 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-model-cache\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.364851 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.364782 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f12f7d-97f3-463b-b84f-32129807de15-tls-certs\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.365028 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.365010 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgfk4\" (UniqueName: \"kubernetes.io/projected/52f12f7d-97f3-463b-b84f-32129807de15-kube-api-access-qgfk4\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.365065 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.365053 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-tmp-dir\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.365140 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.365128 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-home\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.466153 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.466111 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgfk4\" (UniqueName: \"kubernetes.io/projected/52f12f7d-97f3-463b-b84f-32129807de15-kube-api-access-qgfk4\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.466354 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.466169 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-tmp-dir\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.466354 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.466205 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-home\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.466354 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.466237 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-dshm\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.466354 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.466271 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-model-cache\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.466354 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.466304 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f12f7d-97f3-463b-b84f-32129807de15-tls-certs\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.466708 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.466682 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-tmp-dir\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.466789 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.466709 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-model-cache\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.466789 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.466728 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-home\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.468550 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.468529 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-dshm\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.468894 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.468865 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f12f7d-97f3-463b-b84f-32129807de15-tls-certs\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.519135 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.519059 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgfk4\" (UniqueName: \"kubernetes.io/projected/52f12f7d-97f3-463b-b84f-32129807de15-kube-api-access-qgfk4\") pod \"premium-w0-093b97-kserve-5c89fcfd48-czvtg\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.581702 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.581666 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:40.946638 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:40.946606 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg"] Oct 31 19:40:40.948801 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:40:40.948770 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52f12f7d_97f3_463b_b84f_32129807de15.slice/crio-dd1a899f02f67617060e32d7faeffd9e7577612d430ff9a529e0bc993e18359f WatchSource:0}: Error finding container dd1a899f02f67617060e32d7faeffd9e7577612d430ff9a529e0bc993e18359f: Status 404 returned error can't find the container with id dd1a899f02f67617060e32d7faeffd9e7577612d430ff9a529e0bc993e18359f Oct 31 19:40:41.492391 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:41.492356 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" event={"ID":"52f12f7d-97f3-463b-b84f-32129807de15","Type":"ContainerStarted","Data":"7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91"} Oct 31 19:40:41.492391 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:41.492397 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" event={"ID":"52f12f7d-97f3-463b-b84f-32129807de15","Type":"ContainerStarted","Data":"dd1a899f02f67617060e32d7faeffd9e7577612d430ff9a529e0bc993e18359f"} Oct 31 19:40:41.582809 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:41.582771 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:41.600876 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:41.600817 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podStartSLOduration=1.600799547 podStartE2EDuration="1.600799547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:41.597678295 +0000 UTC m=+1413.231072202" watchObservedRunningTime="2026-10-31 19:40:41.600799547 +0000 UTC m=+1413.234193455" Oct 31 19:40:50.403916 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:40:50.403865 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:40:50.581874 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:50.581836 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:50.594278 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:50.594243 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:40:51.571477 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:40:51.571444 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:41:06.080864 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.080827 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm"] Oct 31 19:41:06.083837 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.083815 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.181241 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.181207 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"openshift-service-ca.crt\"" Oct 31 19:41:06.181241 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.181220 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"kube-root-ca.crt\"" Oct 31 19:41:06.181450 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.181253 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"default-dockercfg-96g4b\"" Oct 31 19:41:06.181450 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.181275 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"premium-w5-570915-kserve-self-signed-certs\"" Oct 31 19:41:06.184086 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.184061 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm"] Oct 31 19:41:06.254185 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.254154 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-home\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.254417 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.254234 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-model-cache\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.254491 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.254462 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tls-certs\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.254581 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.254556 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tl25\" (UniqueName: \"kubernetes.io/projected/e43bb0c6-0f01-4106-99f8-7a2c8985315f-kube-api-access-7tl25\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.254619 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.254594 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tmp-dir\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.254734 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.254622 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-dshm\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.356171 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.356084 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tls-certs\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.356171 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.356143 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tl25\" (UniqueName: \"kubernetes.io/projected/e43bb0c6-0f01-4106-99f8-7a2c8985315f-kube-api-access-7tl25\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.356171 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.356163 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tmp-dir\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.356457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.356183 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-dshm\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.356457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.356215 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-home\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.356457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.356278 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-model-cache\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.356756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.356721 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-home\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.356850 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.356759 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-model-cache\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.356850 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.356808 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tmp-dir\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.358739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.358704 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tls-certs\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.358936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.358915 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-dshm\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.408718 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.408686 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tl25\" (UniqueName: \"kubernetes.io/projected/e43bb0c6-0f01-4106-99f8-7a2c8985315f-kube-api-access-7tl25\") pod \"premium-w5-570915-kserve-784fc76bf4-m4jvm\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:06.695263 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:06.695171 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:07.062319 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:07.062297 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm"] Oct 31 19:41:07.665652 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:07.665610 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" event={"ID":"e43bb0c6-0f01-4106-99f8-7a2c8985315f","Type":"ContainerStarted","Data":"947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913"} Oct 31 19:41:07.665652 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:07.665660 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" event={"ID":"e43bb0c6-0f01-4106-99f8-7a2c8985315f","Type":"ContainerStarted","Data":"14cb34ddbed40fe10b414937744b82de7dc9a57e7d06970dc6046940cb7c0b16"} Oct 31 19:41:07.696125 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:07.696084 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:07.774378 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:07.774325 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podStartSLOduration=1.774312009 podStartE2EDuration="1.774312009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:07.772442127 +0000 UTC m=+1439.405836034" watchObservedRunningTime="2026-10-31 19:41:07.774312009 +0000 UTC m=+1439.407705916" Oct 31 19:41:16.695643 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:16.695609 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:16.708071 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:16.708042 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:16.739958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:16.739928 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:41:20.404439 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:41:20.404353 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:41:40.246448 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.246412 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4"] Oct 31 19:41:40.249818 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.249798 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.329343 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.329310 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tmp-dir\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.329550 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.329358 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-home\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.329550 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.329486 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tls-certs\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.329550 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.329526 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-model-cache\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.329734 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.329622 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-dshm\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.329734 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.329646 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtmbf\" (UniqueName: \"kubernetes.io/projected/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-kube-api-access-xtmbf\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.345184 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.345153 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-967dee\"/\"premium-w2-967dee-kserve-self-signed-certs\"" Oct 31 19:41:40.351351 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.351317 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4"] Oct 31 19:41:40.430830 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.430799 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tmp-dir\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.431060 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.430847 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-home\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.431060 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.430947 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tls-certs\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.431060 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.430989 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-model-cache\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.431234 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.431078 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-dshm\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.431234 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.431117 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtmbf\" (UniqueName: \"kubernetes.io/projected/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-kube-api-access-xtmbf\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.433365 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.431309 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tmp-dir\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.433365 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.431593 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-home\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.433365 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.431647 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-model-cache\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.434162 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.434138 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-dshm\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.434352 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.434334 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tls-certs\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.484207 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.484173 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtmbf\" (UniqueName: \"kubernetes.io/projected/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-kube-api-access-xtmbf\") pod \"premium-w2-967dee-kserve-54f5c97dc8-kbhb4\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.561548 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.561460 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:40.964158 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:40.964131 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4"] Oct 31 19:41:40.966000 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:41:40.965968 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1702ff79_4331_4a28_9ea8_63cc3d5d5a50.slice/crio-5bcf5aa65bd939c0e073f7994747b3212a486d9fbd7a6e71d721a9469bf9ecbc WatchSource:0}: Error finding container 5bcf5aa65bd939c0e073f7994747b3212a486d9fbd7a6e71d721a9469bf9ecbc: Status 404 returned error can't find the container with id 5bcf5aa65bd939c0e073f7994747b3212a486d9fbd7a6e71d721a9469bf9ecbc Oct 31 19:41:41.879973 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:41.879928 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" event={"ID":"1702ff79-4331-4a28-9ea8-63cc3d5d5a50","Type":"ContainerStarted","Data":"07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e"} Oct 31 19:41:41.879973 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:41.879986 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" event={"ID":"1702ff79-4331-4a28-9ea8-63cc3d5d5a50","Type":"ContainerStarted","Data":"5bcf5aa65bd939c0e073f7994747b3212a486d9fbd7a6e71d721a9469bf9ecbc"} Oct 31 19:41:41.986122 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:41.985976 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podStartSLOduration=1.985956898 podStartE2EDuration="1.985956898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:41.985521727 +0000 UTC m=+1473.618915637" watchObservedRunningTime="2026-10-31 19:41:41.985956898 +0000 UTC m=+1473.619350804" Oct 31 19:41:42.561562 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:42.561526 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:50.402794 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:41:50.402761 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:41:50.562176 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:50.562149 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:50.577002 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:50.576974 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:41:50.948987 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:41:50.948960 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:42:02.379729 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.379689 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9"] Oct 31 19:42:02.384150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.384125 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.432114 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.432083 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7cfafd-openshift-default-dockercfg-96gwp\"" Oct 31 19:42:02.482556 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.482524 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.482758 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.482564 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.482758 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.482593 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhbz\" (UniqueName: \"kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-kube-api-access-cwhbz\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.482758 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.482708 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.482758 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.482743 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.482996 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.482768 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.482996 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.482847 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.482996 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.482905 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.483145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.483036 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.487997 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.487974 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9"] Oct 31 19:42:02.584592 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.584562 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.584756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.584646 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.584756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.584674 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.584756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.584711 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhbz\" (UniqueName: \"kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-kube-api-access-cwhbz\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.584756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.584737 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.584978 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.584768 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.584978 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.584798 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.585082 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.584871 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.585082 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.585050 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.585224 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.585201 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.585489 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.585465 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.585572 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.585507 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.585572 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.585514 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.585767 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.585748 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.587571 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.587543 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.587779 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.587762 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.638293 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.638230 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.638293 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.638282 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhbz\" (UniqueName: \"kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-kube-api-access-cwhbz\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:02.698244 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:02.698211 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:03.121214 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:03.121175 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9"] Oct 31 19:42:03.157552 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:42:03.157521 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf6ba13_4bc6_48f3_8f9c_535a43d43738.slice/crio-b14432e73537251ec167dd8aea1dd8f6b6c0fb32790d527468d9fb2c49509ede WatchSource:0}: Error finding container b14432e73537251ec167dd8aea1dd8f6b6c0fb32790d527468d9fb2c49509ede: Status 404 returned error can't find the container with id b14432e73537251ec167dd8aea1dd8f6b6c0fb32790d527468d9fb2c49509ede Oct 31 19:42:03.159870 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:03.159838 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:42:03.159976 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:03.159930 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:42:03.159976 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:03.159960 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:42:04.030611 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:04.030573 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" event={"ID":"bdf6ba13-4bc6-48f3-8f9c-535a43d43738","Type":"ContainerStarted","Data":"e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3"} Oct 31 19:42:04.031036 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:04.030632 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" event={"ID":"bdf6ba13-4bc6-48f3-8f9c-535a43d43738","Type":"ContainerStarted","Data":"b14432e73537251ec167dd8aea1dd8f6b6c0fb32790d527468d9fb2c49509ede"} Oct 31 19:42:04.187232 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:04.187176 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" podStartSLOduration=2.1871599489999998 podStartE2EDuration="2.187159949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:04.185313664 +0000 UTC m=+1495.818707573" watchObservedRunningTime="2026-10-31 19:42:04.187159949 +0000 UTC m=+1495.820553856" Oct 31 19:42:04.699262 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:04.699232 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:04.720904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:04.713825 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:42:04.720904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:04.713921 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" podUID="bdf6ba13-4bc6-48f3-8f9c-535a43d43738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 19:42:05.699207 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:05.699174 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:42:05.699597 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:05.699244 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" podUID="bdf6ba13-4bc6-48f3-8f9c-535a43d43738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 19:42:06.699197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:06.699159 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:42:06.699392 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:06.699210 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" podUID="bdf6ba13-4bc6-48f3-8f9c-535a43d43738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 19:42:07.698952 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:07.698917 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:42:07.699130 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:07.698976 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" podUID="bdf6ba13-4bc6-48f3-8f9c-535a43d43738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 19:42:08.698840 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:08.698802 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:42:08.699336 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:08.698856 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" podUID="bdf6ba13-4bc6-48f3-8f9c-535a43d43738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 19:42:09.703298 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:09.703268 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:09.703765 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:09.703740 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:09.704535 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:42:09.704516 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 19:42:20.403948 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:42:20.403907 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:42:50.403701 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:42:50.403616 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:43:01.866401 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.866367 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh"] Oct 31 19:43:01.870377 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.870356 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:01.967562 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.967530 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-093b97-openshift-default-dockercfg-mndxw\"" Oct 31 19:43:01.970212 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.970189 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh"] Oct 31 19:43:01.988458 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.988429 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:01.988637 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.988484 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjcnw\" (UniqueName: \"kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-kube-api-access-zjcnw\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:01.988637 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.988609 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:01.988738 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.988656 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:01.988738 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.988702 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4507c2-4259-4de9-91d6-49329d702cca-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:01.988860 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.988845 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:01.988946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.988876 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:01.988993 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.988966 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4507c2-4259-4de9-91d6-49329d702cca-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:01.989030 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:01.988990 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.089739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.089699 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.089739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.089746 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.090065 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.089799 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4507c2-4259-4de9-91d6-49329d702cca-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.090065 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.089913 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.090065 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.090001 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.090065 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.090047 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjcnw\" (UniqueName: \"kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-kube-api-access-zjcnw\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.090278 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.090106 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.090278 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.090127 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.090278 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.090148 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4507c2-4259-4de9-91d6-49329d702cca-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.090278 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.090232 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.090491 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.090462 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.090597 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.090575 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.090661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.090602 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4507c2-4259-4de9-91d6-49329d702cca-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.090661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.090621 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.092147 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.092128 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.092480 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.092463 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4507c2-4259-4de9-91d6-49329d702cca-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.142757 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.142695 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.143048 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.143025 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjcnw\" (UniqueName: \"kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-kube-api-access-zjcnw\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.184366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.184331 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:02.598541 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.598514 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh"] Oct 31 19:43:02.600517 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:43:02.600488 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4507c2_4259_4de9_91d6_49329d702cca.slice/crio-dd20675fc864f3e27d99265324efc49806ed81d0064113644b22a48702ac718f WatchSource:0}: Error finding container dd20675fc864f3e27d99265324efc49806ed81d0064113644b22a48702ac718f: Status 404 returned error can't find the container with id dd20675fc864f3e27d99265324efc49806ed81d0064113644b22a48702ac718f Oct 31 19:43:02.603071 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.603016 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:43:02.603179 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.603105 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:43:02.603179 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:02.603159 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:43:03.445050 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:03.444829 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" event={"ID":"4d4507c2-4259-4de9-91d6-49329d702cca","Type":"ContainerStarted","Data":"d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026"} Oct 31 19:43:03.445671 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:03.445636 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" event={"ID":"4d4507c2-4259-4de9-91d6-49329d702cca","Type":"ContainerStarted","Data":"dd20675fc864f3e27d99265324efc49806ed81d0064113644b22a48702ac718f"} Oct 31 19:43:03.622903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:03.608679 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" podStartSLOduration=2.608657997 podStartE2EDuration="2.608657997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:03.604315731 +0000 UTC m=+1555.237709652" watchObservedRunningTime="2026-10-31 19:43:03.608657997 +0000 UTC m=+1555.242051906" Oct 31 19:43:04.185562 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:04.185526 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:04.187310 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:04.187277 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:04.187448 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:04.187341 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" podUID="4d4507c2-4259-4de9-91d6-49329d702cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 19:43:05.185305 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:05.185271 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:05.185699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:05.185348 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" podUID="4d4507c2-4259-4de9-91d6-49329d702cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 19:43:06.184896 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:06.184854 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:06.185046 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:06.184935 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" podUID="4d4507c2-4259-4de9-91d6-49329d702cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 19:43:07.184792 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:07.184758 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:07.185237 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:07.184812 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" podUID="4d4507c2-4259-4de9-91d6-49329d702cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 19:43:08.184951 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:08.184916 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:08.185350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:08.184964 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" podUID="4d4507c2-4259-4de9-91d6-49329d702cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 19:43:09.230937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:09.230870 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:09.231564 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:09.231542 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:09.232511 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:09.232483 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:43:16.102960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.102927 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn"] Oct 31 19:43:16.107006 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.106989 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.156380 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.156354 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5a4438-openshift-default-dockercfg-7kvr6\"" Oct 31 19:43:16.192464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.192434 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e32d79-e9ef-4b86-830c-2e1726c09b81-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.192641 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.192481 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.192641 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.192610 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bswh\" (UniqueName: \"kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-kube-api-access-2bswh\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.192777 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.192641 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.192777 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.192739 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.192895 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.192819 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.192968 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.192928 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.192968 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.192958 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.193078 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.193029 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.205015 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.204987 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn"] Oct 31 19:43:16.294682 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.294642 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.294682 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.294683 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.294925 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.294719 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.294925 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.294760 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e32d79-e9ef-4b86-830c-2e1726c09b81-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.295015 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.294924 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.295074 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.295052 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bswh\" (UniqueName: \"kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-kube-api-access-2bswh\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.295127 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.295068 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.295127 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.295095 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.295224 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.295122 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.295224 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.295198 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.295315 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.295221 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.295315 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.295239 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.295411 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.295390 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.295499 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.295480 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e32d79-e9ef-4b86-830c-2e1726c09b81-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.297109 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.297088 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.297394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.297377 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.353145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.353077 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bswh\" (UniqueName: \"kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-kube-api-access-2bswh\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.355516 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.355489 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.419930 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.419869 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:16.889512 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.889477 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn"] Oct 31 19:43:16.893919 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:43:16.893874 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e32d79_e9ef_4b86_830c_2e1726c09b81.slice/crio-861de40e4c678f2941d2d01e11b7cc2bb68c32d91e15e44cfa0ff25ca087054e WatchSource:0}: Error finding container 861de40e4c678f2941d2d01e11b7cc2bb68c32d91e15e44cfa0ff25ca087054e: Status 404 returned error can't find the container with id 861de40e4c678f2941d2d01e11b7cc2bb68c32d91e15e44cfa0ff25ca087054e Oct 31 19:43:16.896235 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.896204 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:43:16.896339 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.896277 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:43:16.896339 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:16.896306 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:43:17.558090 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:17.558050 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" event={"ID":"53e32d79-e9ef-4b86-830c-2e1726c09b81","Type":"ContainerStarted","Data":"44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911"} Oct 31 19:43:17.558624 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:17.558145 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" event={"ID":"53e32d79-e9ef-4b86-830c-2e1726c09b81","Type":"ContainerStarted","Data":"861de40e4c678f2941d2d01e11b7cc2bb68c32d91e15e44cfa0ff25ca087054e"} Oct 31 19:43:17.736409 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:17.723916 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" podStartSLOduration=1.723872884 podStartE2EDuration="1.723872884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:17.700479778 +0000 UTC m=+1569.333873694" watchObservedRunningTime="2026-10-31 19:43:17.723872884 +0000 UTC m=+1569.357266792" Oct 31 19:43:18.422899 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:18.422845 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:18.423289 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:18.423266 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:18.423472 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:18.423447 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" podUID="53e32d79-e9ef-4b86-830c-2e1726c09b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 19:43:19.420308 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:19.420273 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:19.420920 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:19.420862 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" podUID="53e32d79-e9ef-4b86-830c-2e1726c09b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 19:43:20.428988 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:43:20.428948 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:43:20.429733 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:20.429706 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:20.429821 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:20.429797 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" podUID="53e32d79-e9ef-4b86-830c-2e1726c09b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 19:43:21.423906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:21.422979 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:21.423906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:21.423035 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" podUID="53e32d79-e9ef-4b86-830c-2e1726c09b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 19:43:22.425039 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:22.424994 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:22.425528 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:22.425061 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" podUID="53e32d79-e9ef-4b86-830c-2e1726c09b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 19:43:23.421256 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:23.421219 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:23.421406 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:23.421300 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" podUID="53e32d79-e9ef-4b86-830c-2e1726c09b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 19:43:24.425217 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:24.425185 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:24.425671 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:24.425653 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:24.426418 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:24.426396 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:43:32.473800 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.473755 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh"] Oct 31 19:43:32.478032 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.478010 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.526349 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.526315 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-967dee-openshift-default-dockercfg-btwn6\"" Oct 31 19:43:32.531583 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.531549 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.531733 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.531591 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.531733 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.531680 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.531733 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.531731 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.531933 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.531852 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqnsf\" (UniqueName: \"kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-kube-api-access-tqnsf\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.531989 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.531965 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.532035 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.532003 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.532084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.532036 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.532187 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.532168 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.576786 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.576747 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh"] Oct 31 19:43:32.632948 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.632911 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqnsf\" (UniqueName: \"kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-kube-api-access-tqnsf\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.633145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.632995 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.633145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.633032 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.633145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.633065 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.633145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.633126 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.633362 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.633188 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.633362 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.633212 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.633362 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.633251 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.633362 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.633308 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.633625 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.633510 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.634735 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.634043 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.634735 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.634448 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.634735 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.634688 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.635246 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.635228 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.636382 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.636358 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.636739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.636721 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.689870 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.689837 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqnsf\" (UniqueName: \"kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-kube-api-access-tqnsf\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.691397 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.691373 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:32.792698 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:32.792622 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:33.275792 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:33.275752 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh"] Oct 31 19:43:33.277650 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:43:33.277625 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2c9cc7_732a_4d51_8657_28f8eb60577e.slice/crio-18807e38bd686b03196fbca43c83be3fa5b02467a35ea978da3868d91a6778f8 WatchSource:0}: Error finding container 18807e38bd686b03196fbca43c83be3fa5b02467a35ea978da3868d91a6778f8: Status 404 returned error can't find the container with id 18807e38bd686b03196fbca43c83be3fa5b02467a35ea978da3868d91a6778f8 Oct 31 19:43:33.280138 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:33.280109 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:43:33.280201 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:33.280168 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:43:33.280236 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:33.280197 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:43:33.660923 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:33.660820 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" event={"ID":"5e2c9cc7-732a-4d51-8657-28f8eb60577e","Type":"ContainerStarted","Data":"6114ec6cbfa5fccee38d92ff8713fc0d88015013f8c67b7ea3918032a02bfc9c"} Oct 31 19:43:33.660923 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:33.660875 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" event={"ID":"5e2c9cc7-732a-4d51-8657-28f8eb60577e","Type":"ContainerStarted","Data":"18807e38bd686b03196fbca43c83be3fa5b02467a35ea978da3868d91a6778f8"} Oct 31 19:43:33.775695 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:33.775489 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" podStartSLOduration=1.775471751 podStartE2EDuration="1.775471751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:33.772469961 +0000 UTC m=+1585.405863873" watchObservedRunningTime="2026-10-31 19:43:33.775471751 +0000 UTC m=+1585.408865660" Oct 31 19:43:33.792843 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:33.792813 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:34.794492 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:34.794452 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 19:43:34.794917 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:34.794508 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" podUID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 19:43:35.793548 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:35.793514 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 19:43:35.793749 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:35.793591 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" podUID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 19:43:36.793211 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:36.793177 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 19:43:36.793687 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:36.793239 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" podUID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 19:43:37.793852 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:37.793815 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 19:43:37.794262 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:37.793871 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" podUID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 19:43:38.793437 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:38.793385 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 19:43:38.793650 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:38.793461 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" podUID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 19:43:40.725251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:40.725220 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:40.725766 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:40.725600 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:40.726573 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:43:40.726553 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:43:50.402871 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:43:50.402832 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:44:20.404051 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:44:20.403962 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:44:36.152487 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.152449 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w"] Oct 31 19:44:36.157132 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.157112 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.209529 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.209487 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w"] Oct 31 19:44:36.220252 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.220217 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160c1547-6b7b-4d74-9041-b3eca6e20445-tls-certs\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.220397 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.220268 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-dshm\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.220447 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.220411 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vff7d\" (UniqueName: \"kubernetes.io/projected/160c1547-6b7b-4d74-9041-b3eca6e20445-kube-api-access-vff7d\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.220586 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.220561 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-tmp-dir\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.220638 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.220623 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-home\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.220731 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.220710 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-model-cache\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.249990 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.249964 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-967dee\"/\"distinct-w2-967dee-kserve-self-signed-certs\"" Oct 31 19:44:36.322312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.322268 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-tmp-dir\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.322512 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.322451 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-home\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.322587 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.322524 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-model-cache\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.322679 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.322660 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160c1547-6b7b-4d74-9041-b3eca6e20445-tls-certs\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.322744 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.322701 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-dshm\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.322800 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.322747 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-tmp-dir\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.322800 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.322768 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vff7d\" (UniqueName: \"kubernetes.io/projected/160c1547-6b7b-4d74-9041-b3eca6e20445-kube-api-access-vff7d\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.322800 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.322788 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-home\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.323223 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.323199 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-model-cache\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.325060 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.325030 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-dshm\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.325154 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.325064 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160c1547-6b7b-4d74-9041-b3eca6e20445-tls-certs\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.375331 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.375287 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vff7d\" (UniqueName: \"kubernetes.io/projected/160c1547-6b7b-4d74-9041-b3eca6e20445-kube-api-access-vff7d\") pod \"distinct-w2-967dee-kserve-c55c6f8f-brv8w\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.468938 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.468912 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:36.843316 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:44:36.843259 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod160c1547_6b7b_4d74_9041_b3eca6e20445.slice/crio-c54ef5ce04bf769bbf3abac6437fa1f060365dcb31b4a2baaab63c6e3b96b2d0 WatchSource:0}: Error finding container c54ef5ce04bf769bbf3abac6437fa1f060365dcb31b4a2baaab63c6e3b96b2d0: Status 404 returned error can't find the container with id c54ef5ce04bf769bbf3abac6437fa1f060365dcb31b4a2baaab63c6e3b96b2d0 Oct 31 19:44:36.843628 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:36.843604 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w"] Oct 31 19:44:37.125002 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:37.124915 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" event={"ID":"160c1547-6b7b-4d74-9041-b3eca6e20445","Type":"ContainerStarted","Data":"dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a"} Oct 31 19:44:37.125002 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:37.124963 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" event={"ID":"160c1547-6b7b-4d74-9041-b3eca6e20445","Type":"ContainerStarted","Data":"c54ef5ce04bf769bbf3abac6437fa1f060365dcb31b4a2baaab63c6e3b96b2d0"} Oct 31 19:44:37.231986 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:37.231867 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podStartSLOduration=1.231851954 podStartE2EDuration="1.231851954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:37.229605286 +0000 UTC m=+1648.862999192" watchObservedRunningTime="2026-10-31 19:44:37.231851954 +0000 UTC m=+1648.865245894" Oct 31 19:44:37.469486 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:37.469453 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:46.469188 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:46.469150 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:46.487648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:46.487619 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:47.086565 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.086533 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk"] Oct 31 19:44:47.090671 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.090646 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.166136 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.166108 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk"] Oct 31 19:44:47.208031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.207994 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:44:47.208297 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.208269 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.208397 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.208336 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.208461 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.208428 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.208554 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.208528 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.208608 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.208594 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.208662 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.208630 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.209014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.208733 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.209014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.208769 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736a0b55-5f01-414e-b297-6a1db400b1cf-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.209014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.208790 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rncx\" (UniqueName: \"kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-kube-api-access-6rncx\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.309781 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.309746 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.309971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.309789 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736a0b55-5f01-414e-b297-6a1db400b1cf-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.309971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.309808 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rncx\" (UniqueName: \"kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-kube-api-access-6rncx\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.309971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.309835 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.309971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.309869 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.309971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.309910 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.310260 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.310018 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.310260 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.310098 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.310260 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.310144 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.310260 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.310249 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.310471 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.310395 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.310630 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.310604 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.310717 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.310706 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.310861 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.310839 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736a0b55-5f01-414e-b297-6a1db400b1cf-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.312573 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.312551 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.312869 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.312847 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.364420 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.364340 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.364587 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.364564 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rncx\" (UniqueName: \"kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-kube-api-access-6rncx\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.407437 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.407409 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:47.822644 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.822616 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk"] Oct 31 19:44:47.825030 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:44:47.825005 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736a0b55_5f01_414e_b297_6a1db400b1cf.slice/crio-fd03057ffb856195b9b8977f1072f85a7ca03b590ed316c05ef9a6b4f4fedf9b WatchSource:0}: Error finding container fd03057ffb856195b9b8977f1072f85a7ca03b590ed316c05ef9a6b4f4fedf9b: Status 404 returned error can't find the container with id fd03057ffb856195b9b8977f1072f85a7ca03b590ed316c05ef9a6b4f4fedf9b Oct 31 19:44:47.827331 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.827293 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:44:47.827383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.827355 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:44:47.827423 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:47.827381 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:44:48.198846 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:48.198756 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" event={"ID":"736a0b55-5f01-414e-b297-6a1db400b1cf","Type":"ContainerStarted","Data":"9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4"} Oct 31 19:44:48.199061 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:48.198860 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" event={"ID":"736a0b55-5f01-414e-b297-6a1db400b1cf","Type":"ContainerStarted","Data":"fd03057ffb856195b9b8977f1072f85a7ca03b590ed316c05ef9a6b4f4fedf9b"} Oct 31 19:44:48.310802 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:48.310746 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" podStartSLOduration=1.31073087 podStartE2EDuration="1.31073087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:48.307217524 +0000 UTC m=+1659.940611464" watchObservedRunningTime="2026-10-31 19:44:48.31073087 +0000 UTC m=+1659.944124778" Oct 31 19:44:48.408194 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:48.408155 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:48.409991 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:48.409964 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 19:44:48.410147 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:48.410019 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" podUID="736a0b55-5f01-414e-b297-6a1db400b1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 19:44:49.418906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:49.408247 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 19:44:49.418906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:49.408309 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" podUID="736a0b55-5f01-414e-b297-6a1db400b1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 19:44:50.414906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:50.409267 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 19:44:50.414906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:50.409326 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" podUID="736a0b55-5f01-414e-b297-6a1db400b1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 19:44:50.432905 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:44:50.426191 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:44:51.418902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:51.408218 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 19:44:51.418902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:51.408277 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" podUID="736a0b55-5f01-414e-b297-6a1db400b1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 19:44:52.411907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:52.410030 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 19:44:52.411907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:52.410088 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" podUID="736a0b55-5f01-414e-b297-6a1db400b1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 19:44:53.412905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:53.411984 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 19:44:53.412905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:53.412045 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" podUID="736a0b55-5f01-414e-b297-6a1db400b1cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 19:44:54.725382 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:54.725347 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:54.725936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:54.725868 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:44:54.726704 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:44:54.726680 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:45:00.168019 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.167979 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r"] Oct 31 19:45:00.171929 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.171911 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" Oct 31 19:45:00.184222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.184199 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv"] Oct 31 19:45:00.188067 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.188050 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" Oct 31 19:45:00.220103 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.220069 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 19:45:00.260738 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.260709 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r"] Oct 31 19:45:00.278461 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.278423 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gjch\" (UniqueName: \"kubernetes.io/projected/d4df1983-21bc-4127-b423-c25cd3e4a41d-kube-api-access-5gjch\") pod \"maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r\" (UID: \"d4df1983-21bc-4127-b423-c25cd3e4a41d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" Oct 31 19:45:00.278634 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.278614 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vpc4\" (UniqueName: \"kubernetes.io/projected/cf412404-e129-4f26-b52e-2067ef66b2e6-kube-api-access-7vpc4\") pod \"maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv\" (UID: \"cf412404-e129-4f26-b52e-2067ef66b2e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" Oct 31 19:45:00.278709 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.278679 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf412404-e129-4f26-b52e-2067ef66b2e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv\" (UID: \"cf412404-e129-4f26-b52e-2067ef66b2e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" Oct 31 19:45:00.278811 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.278786 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4df1983-21bc-4127-b423-c25cd3e4a41d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r\" (UID: \"d4df1983-21bc-4127-b423-c25cd3e4a41d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" Oct 31 19:45:00.320274 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.320209 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv"] Oct 31 19:45:00.380419 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.380211 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gjch\" (UniqueName: \"kubernetes.io/projected/d4df1983-21bc-4127-b423-c25cd3e4a41d-kube-api-access-5gjch\") pod \"maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r\" (UID: \"d4df1983-21bc-4127-b423-c25cd3e4a41d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" Oct 31 19:45:00.380611 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.380425 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vpc4\" (UniqueName: \"kubernetes.io/projected/cf412404-e129-4f26-b52e-2067ef66b2e6-kube-api-access-7vpc4\") pod \"maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv\" (UID: \"cf412404-e129-4f26-b52e-2067ef66b2e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" Oct 31 19:45:00.380611 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.380510 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf412404-e129-4f26-b52e-2067ef66b2e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv\" (UID: \"cf412404-e129-4f26-b52e-2067ef66b2e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" Oct 31 19:45:00.380611 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.380552 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4df1983-21bc-4127-b423-c25cd3e4a41d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r\" (UID: \"d4df1983-21bc-4127-b423-c25cd3e4a41d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" Oct 31 19:45:00.381355 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.381322 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf412404-e129-4f26-b52e-2067ef66b2e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv\" (UID: \"cf412404-e129-4f26-b52e-2067ef66b2e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" Oct 31 19:45:00.381472 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.381388 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4df1983-21bc-4127-b423-c25cd3e4a41d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r\" (UID: \"d4df1983-21bc-4127-b423-c25cd3e4a41d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" Oct 31 19:45:00.433141 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.433070 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gjch\" (UniqueName: \"kubernetes.io/projected/d4df1983-21bc-4127-b423-c25cd3e4a41d-kube-api-access-5gjch\") pod \"maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r\" (UID: \"d4df1983-21bc-4127-b423-c25cd3e4a41d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" Oct 31 19:45:00.433304 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.433152 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vpc4\" (UniqueName: \"kubernetes.io/projected/cf412404-e129-4f26-b52e-2067ef66b2e6-kube-api-access-7vpc4\") pod \"maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv\" (UID: \"cf412404-e129-4f26-b52e-2067ef66b2e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" Oct 31 19:45:00.485737 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.485710 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" Oct 31 19:45:00.501025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.500996 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" Oct 31 19:45:00.864994 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.864968 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r"] Oct 31 19:45:00.866326 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:45:00.866302 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4df1983_21bc_4127_b423_c25cd3e4a41d.slice/crio-849efbf37efaa8a27c9b24f9dc8aea422b4c9810b9956670f760f0b017c9a340 WatchSource:0}: Error finding container 849efbf37efaa8a27c9b24f9dc8aea422b4c9810b9956670f760f0b017c9a340: Status 404 returned error can't find the container with id 849efbf37efaa8a27c9b24f9dc8aea422b4c9810b9956670f760f0b017c9a340 Oct 31 19:45:00.868302 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.868287 6107 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:45:00.909105 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:00.909079 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv"] Oct 31 19:45:00.910063 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:45:00.910039 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf412404_e129_4f26_b52e_2067ef66b2e6.slice/crio-cb56f06fd7ff9ecb73c06ee8dcbbf89f375d7de5c4aaeb6f706ad01b7b4db04a WatchSource:0}: Error finding container cb56f06fd7ff9ecb73c06ee8dcbbf89f375d7de5c4aaeb6f706ad01b7b4db04a: Status 404 returned error can't find the container with id cb56f06fd7ff9ecb73c06ee8dcbbf89f375d7de5c4aaeb6f706ad01b7b4db04a Oct 31 19:45:01.318431 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:01.318395 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" event={"ID":"d4df1983-21bc-4127-b423-c25cd3e4a41d","Type":"ContainerStarted","Data":"849efbf37efaa8a27c9b24f9dc8aea422b4c9810b9956670f760f0b017c9a340"} Oct 31 19:45:01.319708 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:01.319677 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" event={"ID":"cf412404-e129-4f26-b52e-2067ef66b2e6","Type":"ContainerStarted","Data":"cb56f06fd7ff9ecb73c06ee8dcbbf89f375d7de5c4aaeb6f706ad01b7b4db04a"} Oct 31 19:45:02.807758 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:02.807728 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897"] Oct 31 19:45:02.813338 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:02.813313 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:02.910182 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:02.910148 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897"] Oct 31 19:45:02.910626 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:02.910602 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:02.910670 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:02.910642 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:02.910826 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:02.910802 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:02.910922 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:02.910873 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:02.910974 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:02.910955 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:02.911014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:02.911004 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:02.911150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:02.911136 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:02.911228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:02.911213 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7bhj\" (UniqueName: \"kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-kube-api-access-z7bhj\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:02.911292 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:02.911263 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616e72de-9485-4db6-96e9-bfc30b9f639a-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.012476 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.012439 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bhj\" (UniqueName: \"kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-kube-api-access-z7bhj\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.012476 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.012484 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616e72de-9485-4db6-96e9-bfc30b9f639a-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.012733 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.012516 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.012733 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.012538 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.012733 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.012578 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.012733 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.012613 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.012733 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.012652 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.012733 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.012692 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.013067 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.012769 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.013130 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.013070 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.013192 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.013172 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.013251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.013227 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.013346 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.013325 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.013409 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.013359 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616e72de-9485-4db6-96e9-bfc30b9f639a-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.014809 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.014790 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.015166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.015138 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.065054 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.064974 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.065350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.065325 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bhj\" (UniqueName: \"kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-kube-api-access-z7bhj\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.127402 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.127363 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:03.563401 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.563375 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897"] Oct 31 19:45:03.564946 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:45:03.564916 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616e72de_9485_4db6_96e9_bfc30b9f639a.slice/crio-f29bb99b9309a9088edf44fcbd671a8bce71522024958cc22fa2973ae08f9eea WatchSource:0}: Error finding container f29bb99b9309a9088edf44fcbd671a8bce71522024958cc22fa2973ae08f9eea: Status 404 returned error can't find the container with id f29bb99b9309a9088edf44fcbd671a8bce71522024958cc22fa2973ae08f9eea Oct 31 19:45:03.567478 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.567441 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:45:03.567564 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.567533 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:45:03.567601 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:03.567577 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:45:04.342292 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:04.342255 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" event={"ID":"616e72de-9485-4db6-96e9-bfc30b9f639a","Type":"ContainerStarted","Data":"61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086"} Oct 31 19:45:04.342775 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:04.342315 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" event={"ID":"616e72de-9485-4db6-96e9-bfc30b9f639a","Type":"ContainerStarted","Data":"f29bb99b9309a9088edf44fcbd671a8bce71522024958cc22fa2973ae08f9eea"} Oct 31 19:45:04.495168 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:04.495095 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" podStartSLOduration=2.49507785 podStartE2EDuration="2.49507785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:04.494127447 +0000 UTC m=+1676.127521355" watchObservedRunningTime="2026-10-31 19:45:04.49507785 +0000 UTC m=+1676.128471759" Oct 31 19:45:05.134360 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:05.134270 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:05.136804 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:05.136546 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:45:05.136804 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:05.136591 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" podUID="616e72de-9485-4db6-96e9-bfc30b9f639a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 19:45:06.131788 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:06.131311 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:45:06.131788 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:06.131374 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" podUID="616e72de-9485-4db6-96e9-bfc30b9f639a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 19:45:07.132982 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:07.132534 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:45:07.132982 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:07.132597 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" podUID="616e72de-9485-4db6-96e9-bfc30b9f639a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 19:45:07.377148 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:07.377112 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" event={"ID":"cf412404-e129-4f26-b52e-2067ef66b2e6","Type":"ContainerStarted","Data":"947f18f79d1c76101e3602d638bb3c841283b9fb5a08e5d322840a60a17b6bde"} Oct 31 19:45:07.488412 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:07.488366 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" podStartSLOduration=2.050044696 podStartE2EDuration="7.488351819s" totalImagesPullingTime="5.438307123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:07.484736439 +0000 UTC m=+1679.118130360" watchObservedRunningTime="2026-10-31 19:45:07.488351819 +0000 UTC m=+1679.121745726" Oct 31 19:45:08.128962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:08.128331 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:45:08.128962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:08.128397 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" podUID="616e72de-9485-4db6-96e9-bfc30b9f639a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 19:45:08.384784 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:08.384699 6107 generic.go:356] "Generic (PLEG): container finished" podID="cf412404-e129-4f26-b52e-2067ef66b2e6" containerID="947f18f79d1c76101e3602d638bb3c841283b9fb5a08e5d322840a60a17b6bde" exitCode=60 Oct 31 19:45:08.385244 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:08.384832 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" event={"ID":"cf412404-e129-4f26-b52e-2067ef66b2e6","Type":"ContainerDied","Data":"947f18f79d1c76101e3602d638bb3c841283b9fb5a08e5d322840a60a17b6bde"} Oct 31 19:45:08.385244 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:08.385120 6107 scope.go:122] "RemoveContainer" containerID="947f18f79d1c76101e3602d638bb3c841283b9fb5a08e5d322840a60a17b6bde" Oct 31 19:45:08.388161 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:08.388137 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" event={"ID":"d4df1983-21bc-4127-b423-c25cd3e4a41d","Type":"ContainerStarted","Data":"9484eda2cd8f449ef84bb76547e175b988b5ad6034c524c3437f546e19a22db3"} Oct 31 19:45:08.596564 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:08.596232 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" podStartSLOduration=2.209313703 podStartE2EDuration="8.596220204s" totalImagesPullingTime="6.386906501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:08.59530273 +0000 UTC m=+1680.228696637" watchObservedRunningTime="2026-10-31 19:45:08.596220204 +0000 UTC m=+1680.229614111" Oct 31 19:45:09.134905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:09.134299 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:45:09.134905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:09.134361 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" podUID="616e72de-9485-4db6-96e9-bfc30b9f639a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 19:45:09.396247 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:09.396155 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" event={"ID":"cf412404-e129-4f26-b52e-2067ef66b2e6","Type":"ContainerStarted","Data":"db2615b400d8b305c240679efbf0dbec0b23f3cbbaec8e00063be6a69504a714"} Oct 31 19:45:09.397964 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:09.397939 6107 generic.go:356] "Generic (PLEG): container finished" podID="d4df1983-21bc-4127-b423-c25cd3e4a41d" containerID="9484eda2cd8f449ef84bb76547e175b988b5ad6034c524c3437f546e19a22db3" exitCode=60 Oct 31 19:45:09.398071 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:09.397986 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" event={"ID":"d4df1983-21bc-4127-b423-c25cd3e4a41d","Type":"ContainerDied","Data":"9484eda2cd8f449ef84bb76547e175b988b5ad6034c524c3437f546e19a22db3"} Oct 31 19:45:09.398277 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:09.398261 6107 scope.go:122] "RemoveContainer" containerID="9484eda2cd8f449ef84bb76547e175b988b5ad6034c524c3437f546e19a22db3" Oct 31 19:45:10.128592 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:10.128500 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:45:10.128592 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:10.128568 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" podUID="616e72de-9485-4db6-96e9-bfc30b9f639a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 19:45:10.408662 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:10.408557 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" event={"ID":"d4df1983-21bc-4127-b423-c25cd3e4a41d","Type":"ContainerStarted","Data":"9c9c87a5cdea6d17d33feff752d186072ad0dd326478cf221dd772bfca2e349f"} Oct 31 19:45:10.410835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:10.410806 6107 generic.go:356] "Generic (PLEG): container finished" podID="cf412404-e129-4f26-b52e-2067ef66b2e6" containerID="db2615b400d8b305c240679efbf0dbec0b23f3cbbaec8e00063be6a69504a714" exitCode=60 Oct 31 19:45:10.410985 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:10.410845 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" event={"ID":"cf412404-e129-4f26-b52e-2067ef66b2e6","Type":"ContainerDied","Data":"db2615b400d8b305c240679efbf0dbec0b23f3cbbaec8e00063be6a69504a714"} Oct 31 19:45:10.410985 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:10.410919 6107 scope.go:122] "RemoveContainer" containerID="947f18f79d1c76101e3602d638bb3c841283b9fb5a08e5d322840a60a17b6bde" Oct 31 19:45:10.411226 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:10.411204 6107 scope.go:122] "RemoveContainer" containerID="db2615b400d8b305c240679efbf0dbec0b23f3cbbaec8e00063be6a69504a714" Oct 31 19:45:10.411569 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:45:10.411545 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv_odh-ai-gateway-infra(cf412404-e129-4f26-b52e-2067ef66b2e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" Oct 31 19:45:11.133499 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:11.133464 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:11.134040 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:11.134016 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:11.134713 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:11.134692 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:45:11.421959 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:11.421243 6107 scope.go:122] "RemoveContainer" containerID="db2615b400d8b305c240679efbf0dbec0b23f3cbbaec8e00063be6a69504a714" Oct 31 19:45:11.421959 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:45:11.421594 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv_odh-ai-gateway-infra(cf412404-e129-4f26-b52e-2067ef66b2e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" Oct 31 19:45:11.428903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:11.425220 6107 generic.go:356] "Generic (PLEG): container finished" podID="d4df1983-21bc-4127-b423-c25cd3e4a41d" containerID="9c9c87a5cdea6d17d33feff752d186072ad0dd326478cf221dd772bfca2e349f" exitCode=60 Oct 31 19:45:11.428903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:11.426305 6107 scope.go:122] "RemoveContainer" containerID="9c9c87a5cdea6d17d33feff752d186072ad0dd326478cf221dd772bfca2e349f" Oct 31 19:45:11.428903 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:45:11.426556 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r_odh-ai-gateway-infra(d4df1983-21bc-4127-b423-c25cd3e4a41d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" Oct 31 19:45:11.428903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:11.426603 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" event={"ID":"d4df1983-21bc-4127-b423-c25cd3e4a41d","Type":"ContainerDied","Data":"9c9c87a5cdea6d17d33feff752d186072ad0dd326478cf221dd772bfca2e349f"} Oct 31 19:45:11.428903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:11.426647 6107 scope.go:122] "RemoveContainer" containerID="9484eda2cd8f449ef84bb76547e175b988b5ad6034c524c3437f546e19a22db3" Oct 31 19:45:12.434039 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:12.434004 6107 scope.go:122] "RemoveContainer" containerID="9c9c87a5cdea6d17d33feff752d186072ad0dd326478cf221dd772bfca2e349f" Oct 31 19:45:12.434442 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:45:12.434293 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r_odh-ai-gateway-infra(d4df1983-21bc-4127-b423-c25cd3e4a41d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" Oct 31 19:45:20.404435 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:45:20.404383 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:45:23.480659 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:23.480629 6107 scope.go:122] "RemoveContainer" containerID="9c9c87a5cdea6d17d33feff752d186072ad0dd326478cf221dd772bfca2e349f" Oct 31 19:45:23.481172 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:23.480708 6107 scope.go:122] "RemoveContainer" containerID="db2615b400d8b305c240679efbf0dbec0b23f3cbbaec8e00063be6a69504a714" Oct 31 19:45:24.515650 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:24.515614 6107 generic.go:356] "Generic (PLEG): container finished" podID="d4df1983-21bc-4127-b423-c25cd3e4a41d" containerID="d9f8d93594ac83e3ff056302a8bb8b5170a98511abc3b7513312325cd8463175" exitCode=60 Oct 31 19:45:24.516105 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:24.515684 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" event={"ID":"d4df1983-21bc-4127-b423-c25cd3e4a41d","Type":"ContainerDied","Data":"d9f8d93594ac83e3ff056302a8bb8b5170a98511abc3b7513312325cd8463175"} Oct 31 19:45:24.516105 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:24.515753 6107 scope.go:122] "RemoveContainer" containerID="9c9c87a5cdea6d17d33feff752d186072ad0dd326478cf221dd772bfca2e349f" Oct 31 19:45:24.516222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:24.516178 6107 scope.go:122] "RemoveContainer" containerID="d9f8d93594ac83e3ff056302a8bb8b5170a98511abc3b7513312325cd8463175" Oct 31 19:45:24.516557 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:45:24.516531 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r_odh-ai-gateway-infra(d4df1983-21bc-4127-b423-c25cd3e4a41d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" Oct 31 19:45:24.517957 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:24.517937 6107 generic.go:356] "Generic (PLEG): container finished" podID="cf412404-e129-4f26-b52e-2067ef66b2e6" containerID="4ee240270182552ff78d5cc3a23d30674d72376d784ed8cbd69b19c1466f4e28" exitCode=60 Oct 31 19:45:24.518046 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:24.517988 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" event={"ID":"cf412404-e129-4f26-b52e-2067ef66b2e6","Type":"ContainerDied","Data":"4ee240270182552ff78d5cc3a23d30674d72376d784ed8cbd69b19c1466f4e28"} Oct 31 19:45:24.518345 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:24.518332 6107 scope.go:122] "RemoveContainer" containerID="4ee240270182552ff78d5cc3a23d30674d72376d784ed8cbd69b19c1466f4e28" Oct 31 19:45:24.518613 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:45:24.518593 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv_odh-ai-gateway-infra(cf412404-e129-4f26-b52e-2067ef66b2e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" Oct 31 19:45:24.529395 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:24.529376 6107 scope.go:122] "RemoveContainer" containerID="db2615b400d8b305c240679efbf0dbec0b23f3cbbaec8e00063be6a69504a714" Oct 31 19:45:25.630725 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:25.630693 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r"] Oct 31 19:45:25.736854 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:25.736811 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv"] Oct 31 19:45:25.804920 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:25.804816 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" Oct 31 19:45:25.893726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:25.893667 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" Oct 31 19:45:25.970646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:25.970614 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4df1983-21bc-4127-b423-c25cd3e4a41d-kube-api-access-5gjch\" (UniqueName: \"kubernetes.io/projected/d4df1983-21bc-4127-b423-c25cd3e4a41d-kube-api-access-5gjch\") pod \"d4df1983-21bc-4127-b423-c25cd3e4a41d\" (UID: \"d4df1983-21bc-4127-b423-c25cd3e4a41d\") " Oct 31 19:45:25.970829 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:25.970709 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4df1983-21bc-4127-b423-c25cd3e4a41d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4df1983-21bc-4127-b423-c25cd3e4a41d-maas-api-service-ca\") pod \"d4df1983-21bc-4127-b423-c25cd3e4a41d\" (UID: \"d4df1983-21bc-4127-b423-c25cd3e4a41d\") " Oct 31 19:45:25.971119 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:25.971092 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4df1983-21bc-4127-b423-c25cd3e4a41d-maas-api-service-ca" pod "d4df1983-21bc-4127-b423-c25cd3e4a41d" (UID: "d4df1983-21bc-4127-b423-c25cd3e4a41d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:45:25.971283 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:25.971264 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4df1983-21bc-4127-b423-c25cd3e4a41d-maas-api-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:25.972920 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:25.972894 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4df1983-21bc-4127-b423-c25cd3e4a41d-kube-api-access-5gjch" pod "d4df1983-21bc-4127-b423-c25cd3e4a41d" (UID: "d4df1983-21bc-4127-b423-c25cd3e4a41d"). InnerVolumeSpecName "kube-api-access-5gjch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:45:26.071761 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.071725 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf412404-e129-4f26-b52e-2067ef66b2e6-kube-api-access-7vpc4\" (UniqueName: \"kubernetes.io/projected/cf412404-e129-4f26-b52e-2067ef66b2e6-kube-api-access-7vpc4\") pod \"cf412404-e129-4f26-b52e-2067ef66b2e6\" (UID: \"cf412404-e129-4f26-b52e-2067ef66b2e6\") " Oct 31 19:45:26.072041 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.071815 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf412404-e129-4f26-b52e-2067ef66b2e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf412404-e129-4f26-b52e-2067ef66b2e6-maas-api-service-ca\") pod \"cf412404-e129-4f26-b52e-2067ef66b2e6\" (UID: \"cf412404-e129-4f26-b52e-2067ef66b2e6\") " Oct 31 19:45:26.072261 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.072237 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf412404-e129-4f26-b52e-2067ef66b2e6-maas-api-service-ca" pod "cf412404-e129-4f26-b52e-2067ef66b2e6" (UID: "cf412404-e129-4f26-b52e-2067ef66b2e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:45:26.072474 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.072459 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf412404-e129-4f26-b52e-2067ef66b2e6-maas-api-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:26.072533 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.072476 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gjch\" (UniqueName: \"kubernetes.io/projected/d4df1983-21bc-4127-b423-c25cd3e4a41d-kube-api-access-5gjch\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:26.074871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.074839 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf412404-e129-4f26-b52e-2067ef66b2e6-kube-api-access-7vpc4" pod "cf412404-e129-4f26-b52e-2067ef66b2e6" (UID: "cf412404-e129-4f26-b52e-2067ef66b2e6"). InnerVolumeSpecName "kube-api-access-7vpc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:45:26.173377 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.173300 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vpc4\" (UniqueName: \"kubernetes.io/projected/cf412404-e129-4f26-b52e-2067ef66b2e6-kube-api-access-7vpc4\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:26.534648 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.534620 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" Oct 31 19:45:26.534822 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.534648 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r" event={"ID":"d4df1983-21bc-4127-b423-c25cd3e4a41d","Type":"ContainerDied","Data":"849efbf37efaa8a27c9b24f9dc8aea422b4c9810b9956670f760f0b017c9a340"} Oct 31 19:45:26.534822 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.534704 6107 scope.go:122] "RemoveContainer" containerID="d9f8d93594ac83e3ff056302a8bb8b5170a98511abc3b7513312325cd8463175" Oct 31 19:45:26.536469 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.536448 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" Oct 31 19:45:26.536633 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.536601 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv" event={"ID":"cf412404-e129-4f26-b52e-2067ef66b2e6","Type":"ContainerDied","Data":"cb56f06fd7ff9ecb73c06ee8dcbbf89f375d7de5c4aaeb6f706ad01b7b4db04a"} Oct 31 19:45:26.544366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.544352 6107 scope.go:122] "RemoveContainer" containerID="4ee240270182552ff78d5cc3a23d30674d72376d784ed8cbd69b19c1466f4e28" Oct 31 19:45:26.687241 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.687212 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r"] Oct 31 19:45:26.690801 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.690776 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891265-9j47r"] Oct 31 19:45:26.795860 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.795820 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv"] Oct 31 19:45:26.801143 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:26.801116 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891265-gd7cv"] Oct 31 19:45:28.487779 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:28.487750 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" path="/var/lib/kubelet/pods/cf412404-e129-4f26-b52e-2067ef66b2e6/volumes" Oct 31 19:45:28.488201 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:28.488186 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" path="/var/lib/kubelet/pods/d4df1983-21bc-4127-b423-c25cd3e4a41d/volumes" Oct 31 19:45:32.123469 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.123431 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m"] Oct 31 19:45:32.124261 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124241 6107 cpu_manager.go:414] "containerMap: removing container" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" containerName="cleanup" Oct 31 19:45:32.124344 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124264 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" containerName="cleanup" Oct 31 19:45:32.124344 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124285 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" containerName="cleanup" Oct 31 19:45:32.124344 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124294 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" containerName="cleanup" Oct 31 19:45:32.124344 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124316 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" containerName="cleanup" Oct 31 19:45:32.124344 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124324 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" containerName="cleanup" Oct 31 19:45:32.124344 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124339 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" containerName="cleanup" Oct 31 19:45:32.124516 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124347 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" containerName="cleanup" Oct 31 19:45:32.124516 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124382 6107 cpu_manager.go:414] "containerMap: removing container" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" containerName="cleanup" Oct 31 19:45:32.124516 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124389 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" containerName="cleanup" Oct 31 19:45:32.124516 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124503 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" containerName="cleanup" Oct 31 19:45:32.124625 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124519 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" containerName="cleanup" Oct 31 19:45:32.124625 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124531 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" containerName="cleanup" Oct 31 19:45:32.124625 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.124540 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" containerName="cleanup" Oct 31 19:45:32.128718 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.128695 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.142871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.142847 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.142995 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.142967 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tzqj\" (UniqueName: \"kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-kube-api-access-5tzqj\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.143045 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.143006 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.143080 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.143044 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.143113 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.143084 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.143184 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.143164 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.143221 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.143201 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.143253 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.143223 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.143308 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.143296 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.221239 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.221203 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m"] Oct 31 19:45:32.244380 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.244346 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.244587 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.244396 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.244797 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.244778 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.244923 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.244816 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.244923 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.244853 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.245044 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.244967 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.245044 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.245032 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.245140 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.245118 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tzqj\" (UniqueName: \"kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-kube-api-access-5tzqj\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.245191 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.245148 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.245191 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.245169 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.245909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.245594 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.245909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.245654 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.245909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.245848 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.246512 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.246453 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.248229 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.248121 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.248229 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.248190 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.297111 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.297082 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.297994 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.297973 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tzqj\" (UniqueName: \"kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-kube-api-access-5tzqj\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.442024 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.441944 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:32.857862 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.857824 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m"] Oct 31 19:45:32.862000 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.861967 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:45:32.862088 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.862045 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:45:32.862135 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:32.862105 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:45:33.586500 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:33.586465 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" event={"ID":"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5","Type":"ContainerStarted","Data":"cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256"} Oct 31 19:45:33.586941 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:33.586528 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" event={"ID":"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5","Type":"ContainerStarted","Data":"4bcd8b282423be6ac411e862c4aa795d0e54fe1fb45e364e813e75301be82e37"} Oct 31 19:45:33.740832 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:33.740782 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" podStartSLOduration=1.740769097 podStartE2EDuration="1.740769097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:33.73911776 +0000 UTC m=+1705.372511690" watchObservedRunningTime="2026-10-31 19:45:33.740769097 +0000 UTC m=+1705.374163005" Oct 31 19:45:34.442377 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:34.442338 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:34.444038 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:34.443997 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:45:34.444204 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:34.444057 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" podUID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 19:45:35.442654 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:35.442614 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:45:35.443057 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:35.442682 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" podUID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 19:45:36.443301 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:36.443255 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:45:36.443710 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:36.443341 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" podUID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 19:45:37.443245 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:37.443204 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:45:37.443450 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:37.443263 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" podUID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 19:45:38.443280 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:38.443235 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:45:38.443493 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:38.443310 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" podUID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 19:45:39.443218 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:39.443179 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:45:39.443409 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:39.443243 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" podUID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 19:45:40.448401 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:40.448374 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:40.448815 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:40.448771 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:40.449681 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:45:40.449659 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 19:45:50.404579 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:45:50.404462 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:46:14.496524 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.496488 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9"] Oct 31 19:46:14.497149 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.497132 6107 cpu_manager.go:414] "containerMap: removing container" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" containerName="cleanup" Oct 31 19:46:14.497149 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.497150 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" containerName="cleanup" Oct 31 19:46:14.497266 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.497257 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf412404-e129-4f26-b52e-2067ef66b2e6" containerName="cleanup" Oct 31 19:46:14.497310 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.497268 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4df1983-21bc-4127-b423-c25cd3e4a41d" containerName="cleanup" Oct 31 19:46:14.502171 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.502149 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.530381 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.530339 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tmp-dir\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.530569 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.530428 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-dshm\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.530569 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.530468 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tls-certs\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.530569 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.530560 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xsfn\" (UniqueName: \"kubernetes.io/projected/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-kube-api-access-2xsfn\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.530750 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.530672 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-home\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.530750 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.530707 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-model-cache\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.596261 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.596232 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"distinct-2-w5-570915-kserve-self-signed-certs\"" Oct 31 19:46:14.599300 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.599276 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9"] Oct 31 19:46:14.631604 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.631566 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tmp-dir\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.631792 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.631636 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-dshm\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.631792 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.631667 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tls-certs\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.631946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.631827 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xsfn\" (UniqueName: \"kubernetes.io/projected/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-kube-api-access-2xsfn\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.632015 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.631979 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-home\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.632076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.632020 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-model-cache\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.632076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.632040 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tmp-dir\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.632305 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.632282 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-model-cache\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.632398 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.632379 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-home\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.651282 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.634758 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-dshm\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.651282 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.636363 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tls-certs\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.685327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.685288 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xsfn\" (UniqueName: \"kubernetes.io/projected/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-kube-api-access-2xsfn\") pod \"distinct-2-w5-570915-kserve-5d567ffb86-p87n9\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:14.817554 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:14.817465 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:15.252117 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:15.252080 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9"] Oct 31 19:46:15.253860 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:46:15.253833 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a9b87de_cba8_4ff6_ab19_9e1ce089feaf.slice/crio-f7dcbc30485f92adbacedbba2e573b7f7035c32fb23a60a6576ba48b30d715dc WatchSource:0}: Error finding container f7dcbc30485f92adbacedbba2e573b7f7035c32fb23a60a6576ba48b30d715dc: Status 404 returned error can't find the container with id f7dcbc30485f92adbacedbba2e573b7f7035c32fb23a60a6576ba48b30d715dc Oct 31 19:46:15.937093 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:15.937059 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" event={"ID":"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf","Type":"ContainerStarted","Data":"0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92"} Oct 31 19:46:15.937554 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:15.937105 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" event={"ID":"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf","Type":"ContainerStarted","Data":"f7dcbc30485f92adbacedbba2e573b7f7035c32fb23a60a6576ba48b30d715dc"} Oct 31 19:46:16.043434 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:16.043376 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podStartSLOduration=2.043362163 podStartE2EDuration="2.043362163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:16.042276912 +0000 UTC m=+1747.675670820" watchObservedRunningTime="2026-10-31 19:46:16.043362163 +0000 UTC m=+1747.676756070" Oct 31 19:46:16.817581 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:16.817540 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:17.463421 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.463376 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r"] Oct 31 19:46:17.467601 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.467580 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.571456 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.571415 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.571456 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.571451 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.571699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.571476 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.571699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.571598 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.571699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.571685 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.572699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.572311 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.572699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.572344 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.572699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.572385 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbmh9\" (UniqueName: \"kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-kube-api-access-fbmh9\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.572699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.572493 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.673936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.673866 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.674132 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.673976 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.674132 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.673995 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.674132 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.674016 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.674132 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.674064 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.674132 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.674125 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.674399 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.674233 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.674399 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.674258 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.674399 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.674301 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbmh9\" (UniqueName: \"kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-kube-api-access-fbmh9\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.674555 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.674423 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.674555 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.674457 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.674695 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.674675 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.674895 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.674848 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.675027 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.674981 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.676753 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.676724 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.677048 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.677031 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.686139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.686114 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r"] Oct 31 19:46:17.727935 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.727906 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.728096 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.727996 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbmh9\" (UniqueName: \"kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-kube-api-access-fbmh9\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:17.783120 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:17.783078 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:18.208580 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:18.208552 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r"] Oct 31 19:46:18.209970 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:46:18.209941 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7d07ec_a5e8_4a93_b7d4_9a5764e73a79.slice/crio-43c30ed80d5f12aa8e7c001bc12d1df7b718222b61c5a271b4aeeb64bc60944d WatchSource:0}: Error finding container 43c30ed80d5f12aa8e7c001bc12d1df7b718222b61c5a271b4aeeb64bc60944d: Status 404 returned error can't find the container with id 43c30ed80d5f12aa8e7c001bc12d1df7b718222b61c5a271b4aeeb64bc60944d Oct 31 19:46:18.212520 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:18.212493 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:46:18.212583 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:18.212555 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:46:18.212635 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:18.212585 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:46:18.969653 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:18.969610 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" event={"ID":"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79","Type":"ContainerStarted","Data":"d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58"} Oct 31 19:46:18.970094 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:18.969706 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" event={"ID":"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79","Type":"ContainerStarted","Data":"43c30ed80d5f12aa8e7c001bc12d1df7b718222b61c5a271b4aeeb64bc60944d"} Oct 31 19:46:19.178308 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:19.178254 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" podStartSLOduration=2.178238602 podStartE2EDuration="2.178238602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:19.129656171 +0000 UTC m=+1750.763050080" watchObservedRunningTime="2026-10-31 19:46:19.178238602 +0000 UTC m=+1750.811632508" Oct 31 19:46:19.783385 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:19.783345 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:19.785442 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:19.785417 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:46:19.785572 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:19.785463 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" podUID="ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 19:46:20.471386 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:46:20.471232 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:46:20.784000 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:20.783914 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:46:20.784000 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:20.783983 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" podUID="ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 19:46:21.784182 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:21.784148 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:46:21.784593 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:21.784207 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" podUID="ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 19:46:22.783555 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:22.783517 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:46:22.783773 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:22.783598 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" podUID="ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 19:46:23.784048 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:23.784015 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:46:23.784470 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:23.784068 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" podUID="ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 19:46:24.784471 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:24.784438 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:46:24.784852 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:24.784490 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" podUID="ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 19:46:24.818370 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:24.818335 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:24.833048 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:24.833016 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:25.030832 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:25.030798 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:46:26.440238 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:26.440202 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:26.440670 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:26.440617 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:26.441530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:26.441512 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 19:46:38.617486 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.617445 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc"] Oct 31 19:46:38.621624 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.621601 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.716211 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.716175 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-093b97\"/\"distinct-2-w0-093b97-kserve-self-signed-certs\"" Oct 31 19:46:38.719315 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.719290 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc"] Oct 31 19:46:38.741901 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.741854 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-model-cache\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.742080 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.741921 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-tmp-dir\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.742080 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.741997 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-home\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.742080 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.742028 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58f9cd11-9c06-442c-80fe-40637909f825-tls-certs\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.742080 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.742065 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zftfc\" (UniqueName: \"kubernetes.io/projected/58f9cd11-9c06-442c-80fe-40637909f825-kube-api-access-zftfc\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.742295 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.742199 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-dshm\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.843754 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.843721 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-model-cache\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.843754 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.843759 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-tmp-dir\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.844038 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.843780 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-home\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.844038 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.843796 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58f9cd11-9c06-442c-80fe-40637909f825-tls-certs\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.844038 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.843835 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftfc\" (UniqueName: \"kubernetes.io/projected/58f9cd11-9c06-442c-80fe-40637909f825-kube-api-access-zftfc\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.844038 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.843928 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-dshm\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.844245 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.844210 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-home\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.844304 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.844244 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-model-cache\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.844304 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.844290 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-tmp-dir\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.846240 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.846205 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-dshm\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.846404 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.846387 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58f9cd11-9c06-442c-80fe-40637909f825-tls-certs\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.897224 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.897140 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftfc\" (UniqueName: \"kubernetes.io/projected/58f9cd11-9c06-442c-80fe-40637909f825-kube-api-access-zftfc\") pod \"distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:38.937753 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:38.937715 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:39.371264 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:39.371235 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc"] Oct 31 19:46:39.372386 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:46:39.372355 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58f9cd11_9c06_442c_80fe_40637909f825.slice/crio-a15bfc4cf8228e1edf9b015738421c681d7303ae1907408bf8dc3654cf7c838b WatchSource:0}: Error finding container a15bfc4cf8228e1edf9b015738421c681d7303ae1907408bf8dc3654cf7c838b: Status 404 returned error can't find the container with id a15bfc4cf8228e1edf9b015738421c681d7303ae1907408bf8dc3654cf7c838b Oct 31 19:46:40.165432 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:40.165390 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" event={"ID":"58f9cd11-9c06-442c-80fe-40637909f825","Type":"ContainerStarted","Data":"9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262"} Oct 31 19:46:40.165827 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:40.165443 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" event={"ID":"58f9cd11-9c06-442c-80fe-40637909f825","Type":"ContainerStarted","Data":"a15bfc4cf8228e1edf9b015738421c681d7303ae1907408bf8dc3654cf7c838b"} Oct 31 19:46:40.290175 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:40.290112 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podStartSLOduration=2.29009353 podStartE2EDuration="2.29009353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:40.273635204 +0000 UTC m=+1771.907029114" watchObservedRunningTime="2026-10-31 19:46:40.29009353 +0000 UTC m=+1771.923487530" Oct 31 19:46:40.938672 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:40.938639 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:48.938072 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:48.938029 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:48.952866 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:48.952838 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:49.239643 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:49.239613 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:46:50.403434 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:46:50.403391 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:46:56.400050 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:46:56.400002 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:03.376540 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.376505 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh"] Oct 31 19:47:03.380946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.380929 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.480367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.480324 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh"] Oct 31 19:47:03.553974 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.553941 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.553974 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.553977 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7ln\" (UniqueName: \"kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-kube-api-access-ph7ln\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.554192 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.554110 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.554192 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.554146 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8691586-d76a-495b-9515-99f46a3e7040-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.554258 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.554237 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.554293 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.554280 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.554415 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.554385 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8691586-d76a-495b-9515-99f46a3e7040-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.554468 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.554458 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.554502 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.554479 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.655660 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.655577 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.655660 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.655657 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8691586-d76a-495b-9515-99f46a3e7040-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.655856 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.655699 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.655856 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.655717 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.655856 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.655773 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.655856 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.655791 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7ln\" (UniqueName: \"kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-kube-api-access-ph7ln\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.656063 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.655865 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.656063 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.655910 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8691586-d76a-495b-9515-99f46a3e7040-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.656063 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.655968 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.656063 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.656051 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.656262 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.656108 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.656345 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.656321 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.656423 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.656396 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.656670 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.656652 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8691586-d76a-495b-9515-99f46a3e7040-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.658462 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.658447 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.658774 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.658754 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8691586-d76a-495b-9515-99f46a3e7040-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.708718 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.708681 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7ln\" (UniqueName: \"kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-kube-api-access-ph7ln\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.708924 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.708776 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:03.996002 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:03.995971 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:04.418899 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:04.418857 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh"] Oct 31 19:47:04.421211 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:47:04.421185 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8691586_d76a_495b_9515_99f46a3e7040.slice/crio-12cfdea0b627caff6f2b8897242358fa2c12c439794377bdc696ac860faf21c1 WatchSource:0}: Error finding container 12cfdea0b627caff6f2b8897242358fa2c12c439794377bdc696ac860faf21c1: Status 404 returned error can't find the container with id 12cfdea0b627caff6f2b8897242358fa2c12c439794377bdc696ac860faf21c1 Oct 31 19:47:04.424084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:04.424046 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:04.424169 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:04.424136 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:04.424207 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:04.424179 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:05.371199 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:05.371165 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" event={"ID":"b8691586-d76a-495b-9515-99f46a3e7040","Type":"ContainerStarted","Data":"1e55fd5efa74f3359af678cb8db55a476c6162f41590098bf94d6e0948db9134"} Oct 31 19:47:05.371387 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:05.371217 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" event={"ID":"b8691586-d76a-495b-9515-99f46a3e7040","Type":"ContainerStarted","Data":"12cfdea0b627caff6f2b8897242358fa2c12c439794377bdc696ac860faf21c1"} Oct 31 19:47:05.537786 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:05.537714 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" podStartSLOduration=2.537691892 podStartE2EDuration="2.537691892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:05.537259348 +0000 UTC m=+1797.170653257" watchObservedRunningTime="2026-10-31 19:47:05.537691892 +0000 UTC m=+1797.171085800" Oct 31 19:47:05.996244 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:05.996208 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:05.998168 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:05.998138 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:47:05.998319 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:05.998185 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" podUID="b8691586-d76a-495b-9515-99f46a3e7040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 19:47:06.997257 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:06.997221 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:47:06.997661 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:06.997290 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" podUID="b8691586-d76a-495b-9515-99f46a3e7040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 19:47:07.996816 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:07.996786 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:47:07.997012 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:07.996851 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" podUID="b8691586-d76a-495b-9515-99f46a3e7040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 19:47:08.997301 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:08.997271 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:47:08.997722 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:08.997318 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" podUID="b8691586-d76a-495b-9515-99f46a3e7040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 19:47:09.996929 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:09.996897 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:47:09.997181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:09.996955 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" podUID="b8691586-d76a-495b-9515-99f46a3e7040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 19:47:10.997171 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:10.997131 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:47:10.997685 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:10.997184 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" podUID="b8691586-d76a-495b-9515-99f46a3e7040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 19:47:12.001191 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:12.001159 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:12.001575 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:12.001475 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:12.002337 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:12.002321 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:47:18.226558 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.226478 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96"] Oct 31 19:47:18.231767 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.231748 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.276132 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.276100 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.276271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.276142 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.276271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.276163 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.276347 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.276271 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.276347 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.276289 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.276347 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.276304 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.276472 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.276450 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.276512 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.276486 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3468d78e-548e-41b5-a285-73c62cbe20cc-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.276512 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.276501 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p494r\" (UniqueName: \"kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-kube-api-access-p494r\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.378147 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.378111 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.378329 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.378163 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3468d78e-548e-41b5-a285-73c62cbe20cc-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.378329 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.378185 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p494r\" (UniqueName: \"kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-kube-api-access-p494r\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.378451 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.378366 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.378451 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.378413 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.378586 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.378449 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.378586 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.378485 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.378586 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.378510 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.378586 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.378543 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.378926 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.378822 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.379030 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.379004 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.379352 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.379325 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3468d78e-548e-41b5-a285-73c62cbe20cc-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.379560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.379539 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.379640 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.379616 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.381130 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.381102 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.381491 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.381466 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.421936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.421900 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96"] Oct 31 19:47:18.432121 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.432048 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.438746 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.438695 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p494r\" (UniqueName: \"kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-kube-api-access-p494r\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.553406 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.553322 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:18.829526 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.829447 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp"] Oct 31 19:47:18.835130 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.835105 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.884875 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.884837 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.885158 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.884929 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.885158 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.884972 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.885158 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.885075 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.885158 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.885101 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.885158 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.885124 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.885624 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.885175 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55tl6\" (UniqueName: \"kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-kube-api-access-55tl6\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.885624 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.885214 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.885624 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.885493 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.890368 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.890337 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp"] Oct 31 19:47:18.980238 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.980212 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96"] Oct 31 19:47:18.980946 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:47:18.980918 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3468d78e_548e_41b5_a285_73c62cbe20cc.slice/crio-75e6de91ae44db73d7c39d0e264eff4691839e2a2c46e936e49640cbf91fad9c WatchSource:0}: Error finding container 75e6de91ae44db73d7c39d0e264eff4691839e2a2c46e936e49640cbf91fad9c: Status 404 returned error can't find the container with id 75e6de91ae44db73d7c39d0e264eff4691839e2a2c46e936e49640cbf91fad9c Oct 31 19:47:18.983406 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.983375 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:18.983501 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.983437 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:18.983501 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.983466 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:18.987062 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.987040 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.987199 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.987185 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.987274 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.987225 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.987274 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.987241 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.987388 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.987276 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.987388 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.987343 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.987388 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.987367 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.987523 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.987390 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.987523 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.987426 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55tl6\" (UniqueName: \"kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-kube-api-access-55tl6\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.987523 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.987438 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.987923 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.987871 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.988053 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.987950 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.988920 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.988332 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.989824 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.989799 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.990601 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.990580 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:18.991659 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:18.991639 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:19.040298 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.040270 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55tl6\" (UniqueName: \"kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-kube-api-access-55tl6\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:19.040704 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.040686 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:19.154258 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.154165 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm"] Oct 31 19:47:19.156364 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.156341 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:19.158943 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.158921 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.190016 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.189981 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.190187 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.190045 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.190187 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.190128 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.190292 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.190227 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e2b24f-63f6-40db-aca5-74376cb072f1-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.190354 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.190334 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.190503 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.190487 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5mqd\" (UniqueName: \"kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-kube-api-access-r5mqd\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.190579 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.190554 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.190642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.190628 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.190683 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.190666 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.228841 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.228806 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm"] Oct 31 19:47:19.292646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.291825 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.292646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.291871 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.292646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.291937 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.292646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.292035 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e2b24f-63f6-40db-aca5-74376cb072f1-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.292646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.292091 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.292646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.292117 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5mqd\" (UniqueName: \"kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-kube-api-access-r5mqd\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.292646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.292156 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.292646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.292210 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.292646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.292241 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.293269 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.292756 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.293329 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.293311 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.293607 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.293576 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.293843 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.293823 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e2b24f-63f6-40db-aca5-74376cb072f1-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.293962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.293863 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.295685 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.295657 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.296297 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.296270 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.345901 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.345670 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.346055 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.346027 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5mqd\" (UniqueName: \"kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-kube-api-access-r5mqd\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.443914 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.443814 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7"] Oct 31 19:47:19.449407 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.449381 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.457326 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.457297 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb"] Oct 31 19:47:19.462259 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.462224 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.476328 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.476301 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:19.496542 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.496509 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.496674 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.496565 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.496674 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.496623 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.496674 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.496648 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.496821 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.496680 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.496821 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.496730 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.496821 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.496795 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.497137 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.496831 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.497137 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.496852 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f67579-7b35-47c7-ae72-127726600e78-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.497137 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.496929 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.497137 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.497056 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqpq2\" (UniqueName: \"kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-kube-api-access-rqpq2\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.497137 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.497100 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.497419 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.497148 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.497419 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.497204 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9cnn\" (UniqueName: \"kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-kube-api-access-d9cnn\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.497419 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.497230 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.497419 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.497254 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.497419 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.497269 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.497419 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.497324 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f67579-7b35-47c7-ae72-127726600e78-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.502531 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.502301 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7"] Oct 31 19:47:19.510964 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.510937 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" event={"ID":"3468d78e-548e-41b5-a285-73c62cbe20cc","Type":"ContainerStarted","Data":"5bb6a5a8f766ab851ff05ba8536e5366f21ba7f5f2b8573989f0cbf545c5314f"} Oct 31 19:47:19.511276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.511253 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" event={"ID":"3468d78e-548e-41b5-a285-73c62cbe20cc","Type":"ContainerStarted","Data":"75e6de91ae44db73d7c39d0e264eff4691839e2a2c46e936e49640cbf91fad9c"} Oct 31 19:47:19.521975 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.521909 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb"] Oct 31 19:47:19.554099 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.554069 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:19.567327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.567154 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp"] Oct 31 19:47:19.578405 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:47:19.578366 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc2c4e4_0f1c_4eeb_ac5a_a90a6dc0d0ee.slice/crio-9a75d50b06dbbeefe892f6bc03685be189d9b250fda6bbac19df7a8acb82e357 WatchSource:0}: Error finding container 9a75d50b06dbbeefe892f6bc03685be189d9b250fda6bbac19df7a8acb82e357: Status 404 returned error can't find the container with id 9a75d50b06dbbeefe892f6bc03685be189d9b250fda6bbac19df7a8acb82e357 Oct 31 19:47:19.583033 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.582991 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:19.583129 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.583080 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:19.583129 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.583123 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:19.599054 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.598985 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.599287 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.599036 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.599287 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.599244 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f67579-7b35-47c7-ae72-127726600e78-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.599519 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.599477 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.599752 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.599686 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqpq2\" (UniqueName: \"kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-kube-api-access-rqpq2\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.599848 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.599732 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.600206 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.599990 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.600206 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.600053 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9cnn\" (UniqueName: \"kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-kube-api-access-d9cnn\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.600206 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.600087 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.600206 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.600125 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.600206 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.600148 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.600586 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.600494 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f67579-7b35-47c7-ae72-127726600e78-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.600586 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.600542 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.601297 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.600706 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.601297 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.600773 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.601297 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.600803 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.601297 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.600834 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.601297 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.600919 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.602553 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.601981 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.602553 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.602205 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.603067 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.603048 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.615155 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.603409 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.615320 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.604720 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.615507 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.606424 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f67579-7b35-47c7-ae72-127726600e78-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.615782 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.606659 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.615947 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.607206 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.616049 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.607622 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.616131 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.607795 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.620931 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.620796 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f67579-7b35-47c7-ae72-127726600e78-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.621205 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.621182 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.622351 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.622332 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.622680 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.622646 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.657798 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.657748 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.659228 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.659204 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.659686 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.659663 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9cnn\" (UniqueName: \"kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-kube-api-access-d9cnn\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.666022 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.666003 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqpq2\" (UniqueName: \"kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-kube-api-access-rqpq2\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.678086 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.678042 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podStartSLOduration=1.678029881 podStartE2EDuration="1.678029881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:19.673499415 +0000 UTC m=+1811.306893323" watchObservedRunningTime="2026-10-31 19:47:19.678029881 +0000 UTC m=+1811.311423900" Oct 31 19:47:19.770331 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.770298 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:19.781577 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.781531 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:19.887637 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.887611 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm"] Oct 31 19:47:19.895055 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.895014 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:19.895164 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.895114 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:19.895164 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:19.895156 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:20.163927 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.163896 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7"] Oct 31 19:47:20.166819 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:47:20.166790 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f67579_7b35_47c7_ae72_127726600e78.slice/crio-85bcf6731282dfc33d81fb4ed64acb16a425fc20d7419f01d23ebfa28b6ca202 WatchSource:0}: Error finding container 85bcf6731282dfc33d81fb4ed64acb16a425fc20d7419f01d23ebfa28b6ca202: Status 404 returned error can't find the container with id 85bcf6731282dfc33d81fb4ed64acb16a425fc20d7419f01d23ebfa28b6ca202 Oct 31 19:47:20.170079 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.170044 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:20.170173 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.170113 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:20.170173 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.170141 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:20.183752 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.183729 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb"] Oct 31 19:47:20.184476 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:47:20.184444 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbd00aa_2a24_4e6c_bba8_2b66287efede.slice/crio-acf38485eaf14844e785945a024d44468d856ff032b14274fd3899c974c16260 WatchSource:0}: Error finding container acf38485eaf14844e785945a024d44468d856ff032b14274fd3899c974c16260: Status 404 returned error can't find the container with id acf38485eaf14844e785945a024d44468d856ff032b14274fd3899c974c16260 Oct 31 19:47:20.187200 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.187165 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:20.187287 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.187244 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:20.187287 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.187284 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:20.492457 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:47:20.491264 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:47:20.555294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.555259 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:47:20.555489 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.555312 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 19:47:20.557650 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.556495 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" event={"ID":"a6e2b24f-63f6-40db-aca5-74376cb072f1","Type":"ContainerStarted","Data":"f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f"} Oct 31 19:47:20.557650 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.556555 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" event={"ID":"a6e2b24f-63f6-40db-aca5-74376cb072f1","Type":"ContainerStarted","Data":"7c6eabcbcd231e35356a6ed2f7bb34cf60f6fc6132dfe29a714cd647405d0f9e"} Oct 31 19:47:20.560430 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.560396 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" event={"ID":"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee","Type":"ContainerStarted","Data":"1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413"} Oct 31 19:47:20.560622 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.560606 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" event={"ID":"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee","Type":"ContainerStarted","Data":"9a75d50b06dbbeefe892f6bc03685be189d9b250fda6bbac19df7a8acb82e357"} Oct 31 19:47:20.573241 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.573211 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" event={"ID":"70f67579-7b35-47c7-ae72-127726600e78","Type":"ContainerStarted","Data":"9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a"} Oct 31 19:47:20.573466 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.573452 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" event={"ID":"70f67579-7b35-47c7-ae72-127726600e78","Type":"ContainerStarted","Data":"85bcf6731282dfc33d81fb4ed64acb16a425fc20d7419f01d23ebfa28b6ca202"} Oct 31 19:47:20.588716 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.588686 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" event={"ID":"1dbd00aa-2a24-4e6c-bba8-2b66287efede","Type":"ContainerStarted","Data":"e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953"} Oct 31 19:47:20.588876 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.588735 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" event={"ID":"1dbd00aa-2a24-4e6c-bba8-2b66287efede","Type":"ContainerStarted","Data":"acf38485eaf14844e785945a024d44468d856ff032b14274fd3899c974c16260"} Oct 31 19:47:20.688444 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.687655 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podStartSLOduration=1.687637517 podStartE2EDuration="1.687637517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:20.68329309 +0000 UTC m=+1812.316686999" watchObservedRunningTime="2026-10-31 19:47:20.687637517 +0000 UTC m=+1812.321031426" Oct 31 19:47:20.775070 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.774972 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:20.784068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.783992 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:20.794176 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.794112 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podStartSLOduration=1.7940936029999999 podStartE2EDuration="1.794093603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:20.791367412 +0000 UTC m=+1812.424761319" watchObservedRunningTime="2026-10-31 19:47:20.794093603 +0000 UTC m=+1812.427487513" Oct 31 19:47:20.905495 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:20.905436 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podStartSLOduration=1.9054177060000002 podStartE2EDuration="1.905417706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:20.904635083 +0000 UTC m=+1812.538028992" watchObservedRunningTime="2026-10-31 19:47:20.905417706 +0000 UTC m=+1812.538811615" Oct 31 19:47:21.079266 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.079138 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podStartSLOduration=3.079115203 podStartE2EDuration="3.079115203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:21.0273517 +0000 UTC m=+1812.660745611" watchObservedRunningTime="2026-10-31 19:47:21.079115203 +0000 UTC m=+1812.712509113" Oct 31 19:47:21.202348 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.195770 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:21.204908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.204002 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:47:21.204908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.204067 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 19:47:21.488966 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.480824 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:21.488966 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.481064 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:21.488966 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.481128 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 19:47:21.593411 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.593374 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:47:21.594030 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.593996 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 19:47:21.777843 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.777749 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:21.778124 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.778076 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 19:47:21.841905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.829390 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:21.841905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:21.829449 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:22.165969 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.160267 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:47:22.165969 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.160326 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 19:47:22.339032 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.338986 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:22.511380 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.510914 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:22.511380 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.510973 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 19:47:22.560916 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.558421 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:47:22.560916 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.558482 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 19:47:22.588922 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.588617 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:22.734019 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.733963 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:22.783975 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.783307 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:22.783975 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.783369 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 19:47:22.784839 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.784812 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:22.785195 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.785171 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:22.826010 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.825966 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:47:22.948019 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:22.947965 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 19:47:23.040083 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.039975 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:23.041379 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.041242 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 19:47:23.159076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.158722 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:47:23.159076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.158785 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 19:47:23.194904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.193996 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:23.232902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.232524 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:23.239905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.237963 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:23.457984 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.441506 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 31 19:47:23.457984 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.441924 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:47:23.485921 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.480221 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:23.485921 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.480280 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 19:47:23.615739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.594982 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 19:47:23.615739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.595444 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:47:23.722076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.714317 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:47:23.722076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.714378 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 19:47:23.799946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.777476 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:23.799946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.777554 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 19:47:23.799946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.778823 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 19:47:23.799946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.797495 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:47:23.841076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.841021 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:47:23.841780 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.841758 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:47:23.909028 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.896763 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:23.909028 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.896824 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:23.965108 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:23.964518 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 19:47:24.221962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.197593 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:24.223906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.222527 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:47:24.231913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.224088 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 19:47:24.252928 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.232526 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:47:24.252928 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.232764 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:47:24.252928 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.233003 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:24.399959 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.255842 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:24.592329 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.371126 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:47:24.673916 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.612930 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 19:47:24.673916 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.619261 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:24.673916 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.659630 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:47:24.673916 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.672254 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 19:47:24.829903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.818953 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 19:47:24.858960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.838954 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:24.858960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.839030 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 19:47:24.858960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.842211 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:24.858960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.847297 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:24.858960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.847354 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:24.868130 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.868087 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 19:47:24.965911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.957939 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-shbmz" podUID="a78e6484-6c0b-4c24-a693-1b7c55fe70f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Oct 31 19:47:24.975936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:24.967524 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:25.055934 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.042810 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:47:25.185652 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.185546 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 19:47:25.186849 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.186823 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.187030 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.187008 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 19:47:25.205282 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.205227 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:25.206473 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.206428 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:25.235079 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.235032 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 19:47:25.538933 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.532236 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.538933 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.532296 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 19:47:25.554331 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.554282 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.554586 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.554560 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 19:47:25.685922 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.684043 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 19:47:25.711492 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.711463 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:47:25.778906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.742244 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 19:47:25.780217 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.780184 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.780436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.780411 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 19:47:25.827458 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.827354 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:25.863725 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.863673 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:25.958439 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.958403 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:47:25.983111 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.983056 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.983321 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:25.983139 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:26.031907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.030984 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:26.187447 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.187355 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:47:26.243974 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.235794 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 19:47:26.281891 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.281830 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:26.282466 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.282437 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:26.283332 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.283306 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:26.298958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.298923 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:47:26.767906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.740073 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:47:26.767906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.740145 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 19:47:26.767906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.758677 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:26.767906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.758729 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 19:47:26.786906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.774336 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:47:27.031958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.788140 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.031958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.023843 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:27.031958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.797991 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:27.031958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.788180 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:27.031958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.788359 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.031958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.024413 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 19:47:27.031958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.959111 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:27.031958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.869976 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 31 19:47:27.031958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:26.870029 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 19:47:27.204907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.190551 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:27.286056 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.282968 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:27.288113 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.288085 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.288331 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.288307 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 19:47:27.301910 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.301856 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:27.565974 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.543451 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:27.611330 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.611277 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.611573 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.611544 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 19:47:27.675908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.649971 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.675908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.650064 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 19:47:27.731547 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.731501 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 19:47:27.774122 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.774080 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.774302 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.774161 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 19:47:27.795964 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.795873 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:27.798438 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.798410 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.798635 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.798611 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:27.840979 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.832058 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:47:27.979798 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:27.979758 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bmvw" podUID="6246eaed-3a6b-482b-a25d-fc111a84e89b" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.025130 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.025093 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 19:47:28.042399 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.025038 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.063553 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.063519 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:47:28.064696 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.031976 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.065741 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.032437 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.198061 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.197956 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:47:28.198364 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.198289 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 19:47:28.281008 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.280952 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.281765 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.281743 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:47:28.338731 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.309937 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.338731 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.310306 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:28.478979 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.478366 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:28.478979 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.478422 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 19:47:28.567481 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.567439 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:47:28.567800 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.567774 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 19:47:28.756957 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.752622 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.831936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.801460 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:28.831936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.801517 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:28.977906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.963337 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:28.977906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.963391 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 19:47:28.985311 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:28.985278 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:47:29.044120 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.044001 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:29.072767 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.071980 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:47:29.077065 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.077019 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:29.086912 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.077509 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 19:47:29.320963 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.302960 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:29.320963 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.310008 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:47:29.320963 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.311302 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:47:29.320963 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.311346 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 19:47:29.321448 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.321410 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:29.550095 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.550059 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:29.550328 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.550303 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 19:47:29.782906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.778199 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:47:29.839234 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.792239 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:29.839234 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.792292 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:29.839234 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.792538 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:29.839234 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.792567 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 19:47:29.839234 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.792624 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:29.892905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.890951 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.017904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.944953 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:47:30.017904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.945009 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 19:47:30.017904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:29.973688 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.097963 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.088091 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.097963 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.088471 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.118002 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.117957 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:47:30.305028 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.304974 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.312014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.311962 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.443285 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.443191 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:30.608521 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.605548 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:47:30.608521 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.605618 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 19:47:30.665931 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.664846 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:47:30.665931 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.664923 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 19:47:30.665931 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.665457 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:30.665931 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.665496 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 19:47:30.773144 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.773111 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:30.773413 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.773384 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 19:47:30.809905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.782759 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:30.810204 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.810165 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:30.813532 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.813493 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.909058 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.909006 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.955055 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.955010 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.955652 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:30.955628 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:47:31.102990 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.102143 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:31.225906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.209197 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:31.225906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.212145 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:47:31.334566 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.334523 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:31.335368 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.335342 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:47:31.428117 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.403509 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 19:47:31.476965 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.473236 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:31.477165 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.477066 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:31.477165 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.477121 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 19:47:31.609233 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.598084 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:31.643905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.611250 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:31.788789 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.787377 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded" Oct 31 19:47:31.858771 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.858724 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:32.098280 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.098218 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 19:47:32.098468 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.017954 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.098690 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.903533 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.098906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:31.893061 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:32.102014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.101965 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:32.173441 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.173411 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:47:32.214033 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.213988 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.271054 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.257602 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.369175 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.304999 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.369615 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.346330 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:32.369787 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.369759 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.379023 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.348158 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 19:47:32.380022 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.214260 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 19:47:32.403941 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.403898 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.488374 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.486773 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.724006 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.718990 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:32.724006 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.719055 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.848113 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.839562 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:32.910951 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.843410 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:32.952912 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.940943 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:32.953171 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.939294 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:32.975116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.975029 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:32.989910 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.870681 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 19:47:32.998005 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.979503 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:33.014911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:32.998248 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:33.027570 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.027542 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 19:47:33.027844 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.027826 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:47:33.118916 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.116261 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:33.118916 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.116620 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:33.131381 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.131347 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:47:33.366002 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.352153 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:47:33.367435 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.353680 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Oct 31 19:47:33.410314 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.369547 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:33.424195 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.369592 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 19:47:33.424696 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.398300 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:33.425097 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.409777 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:33.694135 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.693968 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:33.832947 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.809709 6107 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:33.832947 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.809769 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 19:47:33.860946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.860910 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:33.861261 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:33.861231 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:34.119017 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.118976 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:34.190976 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.190932 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 19:47:34.369954 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.367768 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:34.387528 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.387502 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:47:34.411499 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.411466 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:47:34.491619 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.491578 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 19:47:34.512079 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.512031 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 19:47:34.512732 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.512697 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 19:47:34.515223 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.515198 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 19:47:34.709372 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.702916 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:34.869195 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.869166 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:34.869647 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.869620 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:34.869840 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.869821 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:34.869921 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.869867 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:34.871259 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.871242 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:34.871459 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.871446 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:47:34.871920 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.871907 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:47:34.872085 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.872062 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:47:34.872221 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.872211 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:47:34.873606 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.873590 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:47:34.873773 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.873760 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:47:34.900305 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.900279 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d"] Oct 31 19:47:34.910457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.910430 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 19:47:34.910721 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.910690 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d"] Oct 31 19:47:34.910904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.910870 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:34.911665 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.911647 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:47:34.984077 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.984041 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:34.984396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.984367 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:34.984687 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.984661 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:34.984895 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.984839 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:34.985166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.985139 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:34.985408 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.985381 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:34.985701 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.985677 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:34.985799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.985766 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfb5z\" (UniqueName: \"kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-kube-api-access-jfb5z\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:34.986176 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:34.986154 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.087392 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.087351 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.087561 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.087435 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.087561 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.087474 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.087561 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.087523 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.087746 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.087683 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.087798 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.087784 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.087857 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.087821 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfb5z\" (UniqueName: \"kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-kube-api-access-jfb5z\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.087857 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.087839 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.087995 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.087875 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.087995 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.087923 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.087995 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.087975 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.088148 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.087994 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.088236 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.088214 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.088703 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.088679 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.090116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.090090 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.091413 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.090546 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.091945 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.091924 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:35.141954 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.141923 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.142130 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.142007 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfb5z\" (UniqueName: \"kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-kube-api-access-jfb5z\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-wld4d\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.231252 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.231221 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:35.685109 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:47:35.685082 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0ed64f_59a6_4a0a_9293_0d435b303fb5.slice/crio-3eea62097e526d1479a81ca3c341a17a74c41e67b781c986c0978c6f6412713c WatchSource:0}: Error finding container 3eea62097e526d1479a81ca3c341a17a74c41e67b781c986c0978c6f6412713c: Status 404 returned error can't find the container with id 3eea62097e526d1479a81ca3c341a17a74c41e67b781c986c0978c6f6412713c Oct 31 19:47:35.689015 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.688974 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:35.689096 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.689070 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:35.689174 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.689146 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:35.691863 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.691836 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d"] Oct 31 19:47:35.876309 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.876256 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" event={"ID":"dd0ed64f-59a6-4a0a-9293-0d435b303fb5","Type":"ContainerStarted","Data":"d9469f74a1f815d598e3136f38ac346b69fff7455454e769fe2029ddc75f91e0"} Oct 31 19:47:35.876506 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.876319 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" event={"ID":"dd0ed64f-59a6-4a0a-9293-0d435b303fb5","Type":"ContainerStarted","Data":"3eea62097e526d1479a81ca3c341a17a74c41e67b781c986c0978c6f6412713c"} Oct 31 19:47:35.878250 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:35.878194 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:47:36.248930 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:36.244031 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:36.248930 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:36.245683 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:36.248930 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:36.245766 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 19:47:36.631321 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:36.631205 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" podStartSLOduration=2.6311890570000003 podStartE2EDuration="2.631189057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:36.600142515 +0000 UTC m=+1828.233536423" watchObservedRunningTime="2026-10-31 19:47:36.631189057 +0000 UTC m=+1828.264582987" Oct 31 19:47:37.232527 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:37.232494 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:37.232962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:37.232556 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 19:47:38.232514 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:38.232475 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:38.232674 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:38.232548 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 19:47:39.232471 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:39.232435 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:39.232687 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:39.232511 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 19:47:40.232399 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:40.232357 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:40.232546 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:40.232421 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 19:47:41.231958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:41.231922 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:41.232523 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:41.231984 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 19:47:42.231535 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:42.231503 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:42.231701 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:42.231577 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 19:47:43.236590 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:43.236552 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:43.237040 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:43.236912 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:43.237627 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:43.237607 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 19:47:48.782918 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.782865 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj"] Oct 31 19:47:48.787367 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.787344 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.831081 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.831051 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh"] Oct 31 19:47:48.837086 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.837058 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:48.866989 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.866961 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.867138 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.867025 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.867202 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.867149 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.867258 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.867213 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.867314 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.867263 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.867371 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.867320 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.867429 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.867374 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc7pp\" (UniqueName: \"kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-kube-api-access-qc7pp\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.867429 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.867406 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.867528 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.867427 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.876405 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.876370 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj"] Oct 31 19:47:48.887122 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.887091 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh"] Oct 31 19:47:48.969332 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969300 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:48.969520 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969351 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.969520 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969376 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:48.969520 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969510 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:48.969702 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969609 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.969702 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969646 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:48.969785 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969718 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj9g2\" (UniqueName: \"kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-kube-api-access-tj9g2\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:48.969785 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969773 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7pp\" (UniqueName: \"kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-kube-api-access-qc7pp\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.969912 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969803 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:48.969912 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969833 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.969912 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969850 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:48.969912 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969871 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.970116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.969949 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:48.970116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.970000 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:48.970116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.970028 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.970116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.970066 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.970294 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.970171 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.970375 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.970357 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.970516 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.970471 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.970593 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.970512 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.970658 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.970609 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.970859 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.970836 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.970965 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.970947 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.972106 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.972083 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:48.972628 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:48.972608 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:49.022039 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.022009 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:49.022185 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.022079 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc7pp\" (UniqueName: \"kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-kube-api-access-qc7pp\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:49.071845 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.071768 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.071845 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.071815 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.071845 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.071846 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.072153 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.071906 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.072153 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.071971 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj9g2\" (UniqueName: \"kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-kube-api-access-tj9g2\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.072153 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.072009 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.072311 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.072159 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.072311 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.072227 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.072311 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.072293 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.072311 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.072297 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.072490 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.072342 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.072490 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.072368 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.072490 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.072424 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.073393 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.073367 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.074551 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.074530 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.074945 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.074930 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.105087 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.105062 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:49.124600 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.124572 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.124869 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.124848 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj9g2\" (UniqueName: \"kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-kube-api-access-tj9g2\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.162084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.161804 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:49.545105 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.545073 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj"] Oct 31 19:47:49.546078 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:47:49.546051 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a7ba14_180b_4ee3_9917_e13736fd6ad2.slice/crio-8adc1e71ba160a15c691334053f47b9405e8367a17eb6b8c8ad0482222f9e2d1 WatchSource:0}: Error finding container 8adc1e71ba160a15c691334053f47b9405e8367a17eb6b8c8ad0482222f9e2d1: Status 404 returned error can't find the container with id 8adc1e71ba160a15c691334053f47b9405e8367a17eb6b8c8ad0482222f9e2d1 Oct 31 19:47:49.549150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.549115 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:49.549239 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.549200 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:49.549281 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.549238 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:49.589165 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.589140 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh"] Oct 31 19:47:49.590104 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:47:49.590079 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7528b131_3ca2_4da8_a966_bc6a1e8f0d2e.slice/crio-f6d23ab4b96b0491eaffa569962f52585697336b1b1d2c347a9b2cfd1a3e047f WatchSource:0}: Error finding container f6d23ab4b96b0491eaffa569962f52585697336b1b1d2c347a9b2cfd1a3e047f: Status 404 returned error can't find the container with id f6d23ab4b96b0491eaffa569962f52585697336b1b1d2c347a9b2cfd1a3e047f Oct 31 19:47:49.592962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.592930 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:49.593072 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.593019 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:49.593072 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:49.593065 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:50.009024 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.008850 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw"] Oct 31 19:47:50.014683 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.014660 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.015931 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.015819 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" event={"ID":"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e","Type":"ContainerStarted","Data":"0dd0ee4ea2f520ec9e9ba8c60af8032a473ad06920744115659595ce36d58d30"} Oct 31 19:47:50.015931 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.015868 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" event={"ID":"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e","Type":"ContainerStarted","Data":"f6d23ab4b96b0491eaffa569962f52585697336b1b1d2c347a9b2cfd1a3e047f"} Oct 31 19:47:50.018347 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.018295 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" event={"ID":"f9a7ba14-180b-4ee3-9917-e13736fd6ad2","Type":"ContainerStarted","Data":"bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0"} Oct 31 19:47:50.018347 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.018338 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" event={"ID":"f9a7ba14-180b-4ee3-9917-e13736fd6ad2","Type":"ContainerStarted","Data":"8adc1e71ba160a15c691334053f47b9405e8367a17eb6b8c8ad0482222f9e2d1"} Oct 31 19:47:50.098602 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.093164 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.098602 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.093268 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.098602 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.093316 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sfk7\" (UniqueName: \"kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-kube-api-access-8sfk7\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.098602 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.093470 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.098602 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.093542 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.098602 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.093632 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.098602 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.093734 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b26a43c-9759-4e25-a559-adf43a3b0667-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.098602 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.093803 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.098602 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.093865 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.110567 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.107672 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:47:50.110742 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.110646 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:50.110742 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.110728 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:47:50.154752 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.154707 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw"] Oct 31 19:47:50.163120 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.163096 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:47:50.165152 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.165126 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:50.165298 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.165191 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:47:50.195684 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.195656 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.195858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.195707 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.195858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.195742 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b26a43c-9759-4e25-a559-adf43a3b0667-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.195858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.195782 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.195858 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.195827 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.196084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.195923 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.196084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.195959 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.196084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.195998 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sfk7\" (UniqueName: \"kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-kube-api-access-8sfk7\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.196287 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.196104 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.196287 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.196225 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.196388 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.196361 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.196388 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.196373 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.196686 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.196661 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.196789 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.196766 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b26a43c-9759-4e25-a559-adf43a3b0667-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.198101 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.198083 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.198385 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.198368 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.249794 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.249604 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.249794 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.249743 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sfk7\" (UniqueName: \"kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-kube-api-access-8sfk7\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.252067 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.252016 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podStartSLOduration=2.251924449 podStartE2EDuration="2.251924449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:50.251927068 +0000 UTC m=+1841.885320971" watchObservedRunningTime="2026-10-31 19:47:50.251924449 +0000 UTC m=+1841.885318359" Oct 31 19:47:50.346254 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.346153 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:50.362254 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.362202 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podStartSLOduration=2.362187349 podStartE2EDuration="2.362187349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:50.359226716 +0000 UTC m=+1841.992620624" watchObservedRunningTime="2026-10-31 19:47:50.362187349 +0000 UTC m=+1841.995581339" Oct 31 19:47:50.471805 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:47:50.471582 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:47:50.812927 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.807519 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw"] Oct 31 19:47:50.820905 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:47:50.818141 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b26a43c_9759_4e25_a559_adf43a3b0667.slice/crio-5a5a3a4ad7cc5c5cb1088b201a378d0a0923903fd5ca2ceab8dc2a0ee65d4db4 WatchSource:0}: Error finding container 5a5a3a4ad7cc5c5cb1088b201a378d0a0923903fd5ca2ceab8dc2a0ee65d4db4: Status 404 returned error can't find the container with id 5a5a3a4ad7cc5c5cb1088b201a378d0a0923903fd5ca2ceab8dc2a0ee65d4db4 Oct 31 19:47:50.864903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.860960 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:50.864903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.861056 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:50.864903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:50.861099 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:47:51.076258 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:51.076176 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" event={"ID":"6b26a43c-9759-4e25-a559-adf43a3b0667","Type":"ContainerStarted","Data":"79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772"} Oct 31 19:47:51.076763 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:51.076739 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" event={"ID":"6b26a43c-9759-4e25-a559-adf43a3b0667","Type":"ContainerStarted","Data":"5a5a3a4ad7cc5c5cb1088b201a378d0a0923903fd5ca2ceab8dc2a0ee65d4db4"} Oct 31 19:47:51.124906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:51.124433 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:51.124906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:51.124491 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:47:51.176059 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:51.174574 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:51.176059 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:51.174663 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:47:51.190124 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:51.190064 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podStartSLOduration=2.190044998 podStartE2EDuration="2.190044998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:51.184027519 +0000 UTC m=+1842.817421438" watchObservedRunningTime="2026-10-31 19:47:51.190044998 +0000 UTC m=+1842.823438908" Oct 31 19:47:51.357964 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:51.357560 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:47:51.357964 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:51.357700 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:51.357964 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:51.357740 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:47:52.125904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:52.125077 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:52.125904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:52.125142 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:47:52.166905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:52.166584 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:52.166905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:52.166645 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:47:52.347516 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:52.347481 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:52.347801 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:52.347772 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:47:53.193868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:53.192037 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:53.193868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:53.192104 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:47:53.193868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:53.192471 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:53.193868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:53.192496 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:47:53.383536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:53.381445 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:53.383536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:53.381508 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:47:54.052903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:54.051969 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:54.114169 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:54.114133 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:54.114443 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:54.114413 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:47:54.193904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:54.184004 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:54.193904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:54.184063 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:47:54.294910 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:54.293975 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:54.320093 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:54.319982 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:54.370028 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:54.369977 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:54.396216 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:54.396171 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:54.415005 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:54.397254 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:54.415271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:54.415235 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:47:54.900918 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:54.868396 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.035292 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.035238 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.058708 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.058658 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.101907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.097962 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.113901 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.108656 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:47:55.115670 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.115637 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:55.116464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.116436 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:47:55.169963 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.169580 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.204906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.203998 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:55.204906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.204063 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:47:55.340907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.312368 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.409259 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.359092 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:55.413000 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.412959 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:47:55.426030 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.410114 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:47:55.426616 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.367411 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.431031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.416013 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.445214 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.445182 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:47:55.445453 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.445441 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:47:55.446130 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.420704 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.446528 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.446512 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:47:55.932907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.917430 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 19:47:56.063911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:55.934454 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded" Oct 31 19:47:56.064292 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.064260 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:47:56.099932 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.069983 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:56.104910 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.077979 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:56.119066 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.119031 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:47:56.139910 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.136989 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:47:56.139910 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.137418 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:47:56.207967 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.188806 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:56.233003 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.232961 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:47:56.233245 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.225980 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 31 19:47:56.271328 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.228499 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:56.297159 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.297122 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:47:56.297406 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.271588 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:47:56.361351 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.361311 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:56.403039 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.402995 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:47:56.431908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.427979 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 19:47:56.469995 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.432103 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 19:47:56.498909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.432174 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:56.537955 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.459510 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:56.889064 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.878014 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:56.889351 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:56.889314 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" podUID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:57.053420 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.053371 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:57.100594 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.100543 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 19:47:57.137313 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.118025 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 19:47:57.163966 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.119263 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:57.186908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.182970 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:47:57.223905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.150585 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:47:57.256906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.163262 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:57.257202 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.257165 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:47:57.276902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.274405 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 31 19:47:57.495968 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.445449 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 19:47:57.495968 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.491863 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:57.495968 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.491942 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:47:57.495968 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.493288 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:57.507013 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.506964 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:57.535911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.507910 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:57.535911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.507958 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:57.761081 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:57.760977 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 31 19:47:58.060545 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.060436 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:58.064055 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.064019 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:58.064303 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.064269 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" podUID="b8691586-d76a-495b-9515-99f46a3e7040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:58.132906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.124205 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:58.132906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.124266 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:47:58.141310 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.141266 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 19:47:58.152019 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.151971 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 19:47:58.223907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.220282 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:58.223907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.220347 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:47:58.275394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.275345 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:47:58.279077 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.275672 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 31 19:47:58.357123 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.357033 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:58.360986 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.360946 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:47:58.457667 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.456220 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 19:47:58.520905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.515984 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:58.543041 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.538528 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:58.575080 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:58.543098 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 19:47:59.065960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.063066 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:59.202903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.199641 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:59.202903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.199722 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:47:59.202903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.200904 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 19:47:59.202903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.201265 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 19:47:59.209909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.204853 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:59.235905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.225250 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:47:59.344896 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.344770 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:59.345517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.345486 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:59.345713 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.345689 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:59.388029 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.387987 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:59.388302 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.388272 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:47:59.407084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.407042 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:59.468829 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.468769 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 19:47:59.551891 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.551840 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:59.626045 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.615985 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:59.626045 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.616418 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:59.957125 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.957014 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:59.957578 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:47:59.957527 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 31 19:48:00.076344 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.076298 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:00.106391 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.106334 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:48:00.106664 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.106630 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:48:00.164217 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.164181 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:48:00.164411 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.164240 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:48:00.204560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.204515 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:00.208955 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.204810 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:00.410913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.394785 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:00.410913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.407973 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:00.454569 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.454530 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:48:00.454856 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.454825 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:48:00.475620 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.475573 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:00.591130 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.591078 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:00.637907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.637728 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:00.642635 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:00.641967 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 19:48:01.161909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.133763 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:01.161909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.134966 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:48:01.161909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.156005 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:48:01.175330 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.175290 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:48:01.175582 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.175551 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:48:01.207900 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.206846 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:01.231307 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.231264 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:01.346743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.346702 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:48:01.347057 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.347025 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:48:01.406001 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.405957 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:48:01.412074 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.411964 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 31 19:48:01.477010 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.476963 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:01.625917 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.613115 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:01.625917 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.613915 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:48:01.634010 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.633945 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:01.638749 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.637956 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 19:48:01.643001 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.642954 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:01.643330 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.643289 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:01.912199 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.912159 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:48:01.971361 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:01.971333 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:48:02.085908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.085867 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:48:02.111364 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.111332 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:48:02.111682 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.111622 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 19:48:02.143460 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.142791 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 19:48:02.185795 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.178003 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:48:02.185795 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.178054 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 19:48:02.188156 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.187831 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:48:02.207161 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.207131 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:48:02.227292 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.227262 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:48:02.277871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.277841 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:48:02.327469 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.327439 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:48:02.356918 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.351282 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:48:02.356918 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.351331 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:48:02.365123 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:02.365098 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:48:03.360970 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:03.359040 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:48:03.360970 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:03.359101 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:48:03.657576 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:03.657327 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:48:03.658237 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:03.658209 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:48:03.660708 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:03.660685 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:48:03.874524 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:03.874494 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:48:03.874872 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:03.874843 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:48:03.876140 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:03.876116 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:48:04.347141 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:04.347109 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:48:04.347474 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:04.347411 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 19:48:05.348359 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:05.348304 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:48:06.394106 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:06.394069 6107 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:06.394658 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:06.394148 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:06.669860 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:06.669775 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:48:06.671115 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:06.671096 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:48:07.822901 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.822858 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v"] Oct 31 19:48:07.827452 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.827430 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:07.876778 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.876754 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:48:07.881372 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.881349 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v"] Oct 31 19:48:07.892959 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.892929 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4563c9bf-29r6v\" (UID: \"41cf9a87-976b-4d0c-a145-30ce809450d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:07.893150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.893132 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnk2x\" (UniqueName: \"kubernetes.io/projected/41cf9a87-976b-4d0c-a145-30ce809450d0-kube-api-access-jnk2x\") pod \"maas-api-revoke-sub-4563c9bf-29r6v\" (UID: \"41cf9a87-976b-4d0c-a145-30ce809450d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:07.893259 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.893245 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4563c9bf-29r6v\" (UID: \"41cf9a87-976b-4d0c-a145-30ce809450d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:07.921663 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.921628 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 19:48:07.994496 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.994456 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4563c9bf-29r6v\" (UID: \"41cf9a87-976b-4d0c-a145-30ce809450d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:07.994693 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.994611 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4563c9bf-29r6v\" (UID: \"41cf9a87-976b-4d0c-a145-30ce809450d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:07.994762 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.994728 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnk2x\" (UniqueName: \"kubernetes.io/projected/41cf9a87-976b-4d0c-a145-30ce809450d0-kube-api-access-jnk2x\") pod \"maas-api-revoke-sub-4563c9bf-29r6v\" (UID: \"41cf9a87-976b-4d0c-a145-30ce809450d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:07.995172 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.995153 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4563c9bf-29r6v\" (UID: \"41cf9a87-976b-4d0c-a145-30ce809450d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:07.996959 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:07.996932 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4563c9bf-29r6v\" (UID: \"41cf9a87-976b-4d0c-a145-30ce809450d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:08.054457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:08.054425 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnk2x\" (UniqueName: \"kubernetes.io/projected/41cf9a87-976b-4d0c-a145-30ce809450d0-kube-api-access-jnk2x\") pod \"maas-api-revoke-sub-4563c9bf-29r6v\" (UID: \"41cf9a87-976b-4d0c-a145-30ce809450d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:08.141360 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:08.141278 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:08.522769 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:08.522741 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v"] Oct 31 19:48:08.525006 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:48:08.524979 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cf9a87_976b_4d0c_a145_30ce809450d0.slice/crio-e12e4c3915ef1c0afd35f89cee4faf481e76540600735d539893cb5a5a251b5e WatchSource:0}: Error finding container e12e4c3915ef1c0afd35f89cee4faf481e76540600735d539893cb5a5a251b5e: Status 404 returned error can't find the container with id e12e4c3915ef1c0afd35f89cee4faf481e76540600735d539893cb5a5a251b5e Oct 31 19:48:09.460364 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:09.460324 6107 generic.go:356] "Generic (PLEG): container finished" podID="41cf9a87-976b-4d0c-a145-30ce809450d0" containerID="e93eb6e6deb4fbb55ba757f9b60f4ae4df13fb23fa22c22ee69a64fc9202c757" exitCode=0 Oct 31 19:48:09.460741 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:09.460422 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" event={"ID":"41cf9a87-976b-4d0c-a145-30ce809450d0","Type":"ContainerDied","Data":"e93eb6e6deb4fbb55ba757f9b60f4ae4df13fb23fa22c22ee69a64fc9202c757"} Oct 31 19:48:09.460741 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:09.460472 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" event={"ID":"41cf9a87-976b-4d0c-a145-30ce809450d0","Type":"ContainerStarted","Data":"e12e4c3915ef1c0afd35f89cee4faf481e76540600735d539893cb5a5a251b5e"} Oct 31 19:48:10.657080 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:10.657055 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:10.741426 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:10.741389 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-openshift-ca\") pod \"41cf9a87-976b-4d0c-a145-30ce809450d0\" (UID: \"41cf9a87-976b-4d0c-a145-30ce809450d0\") " Oct 31 19:48:10.741715 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:10.741699 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cf9a87-976b-4d0c-a145-30ce809450d0-kube-api-access-jnk2x\" (UniqueName: \"kubernetes.io/projected/41cf9a87-976b-4d0c-a145-30ce809450d0-kube-api-access-jnk2x\") pod \"41cf9a87-976b-4d0c-a145-30ce809450d0\" (UID: \"41cf9a87-976b-4d0c-a145-30ce809450d0\") " Oct 31 19:48:10.741779 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:10.741766 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-xks-ca\") pod \"41cf9a87-976b-4d0c-a145-30ce809450d0\" (UID: \"41cf9a87-976b-4d0c-a145-30ce809450d0\") " Oct 31 19:48:10.741849 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:10.741767 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-openshift-ca" pod "41cf9a87-976b-4d0c-a145-30ce809450d0" (UID: "41cf9a87-976b-4d0c-a145-30ce809450d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:48:10.742500 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:10.742477 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-openshift-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:10.744024 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:10.743998 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cf9a87-976b-4d0c-a145-30ce809450d0-kube-api-access-jnk2x" pod "41cf9a87-976b-4d0c-a145-30ce809450d0" (UID: "41cf9a87-976b-4d0c-a145-30ce809450d0"). InnerVolumeSpecName "kube-api-access-jnk2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:10.744222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:10.744200 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-xks-ca" pod "41cf9a87-976b-4d0c-a145-30ce809450d0" (UID: "41cf9a87-976b-4d0c-a145-30ce809450d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:48:10.843819 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:10.843786 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnk2x\" (UniqueName: \"kubernetes.io/projected/41cf9a87-976b-4d0c-a145-30ce809450d0-kube-api-access-jnk2x\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:10.843819 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:10.843818 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cf9a87-976b-4d0c-a145-30ce809450d0-maas-api-xks-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.477850 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:11.477807 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" event={"ID":"41cf9a87-976b-4d0c-a145-30ce809450d0","Type":"ContainerDied","Data":"e12e4c3915ef1c0afd35f89cee4faf481e76540600735d539893cb5a5a251b5e"} Oct 31 19:48:11.478039 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:11.477823 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v" Oct 31 19:48:11.478039 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:11.477909 6107 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e12e4c3915ef1c0afd35f89cee4faf481e76540600735d539893cb5a5a251b5e" Oct 31 19:48:20.404444 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:48:20.404405 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:48:31.061903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:31.058079 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:31.108130 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:31.108084 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:31.238918 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:31.236789 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:31.316975 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:31.304964 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:31.533931 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:31.525225 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:31.559197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:31.559106 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:48:31.590144 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:31.583004 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:31.634290 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:31.634216 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:32.077906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.070984 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:32.077906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.072053 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:48:32.109935 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.109112 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 19:48:32.109935 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.109698 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:48:32.142916 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.141524 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:32.262908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.262431 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:32.262908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.262793 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:48:32.320907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.306751 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:32.320907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.307202 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:48:32.638907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.638259 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:32.664201 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.664161 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:48:32.680041 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.680000 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 19:48:32.778675 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:32.726803 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 19:48:33.207943 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.205268 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 19:48:33.285305 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.285255 6107 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Oct 31 19:48:33.288863 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.288825 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 19:48:33.410911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.403013 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:33.410911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.404708 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:48:33.411450 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.411417 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:33.549127 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:48:33.549089 6107 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.065s" Oct 31 19:48:33.573326 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.573298 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:48:33.586115 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.574046 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:48:33.586786 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.586759 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:48:33.590751 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.588711 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:48:33.663106 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.663063 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:33.725790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.725748 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:48:33.935957 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.782220 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 19:48:33.935957 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.908972 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:33.935957 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:33.923707 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:48:34.211963 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:34.209644 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:34.316903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:34.300278 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:34.329416 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:34.329382 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:34.350219 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:34.350065 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:34.640016 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:34.625434 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 31 19:48:34.640016 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:34.625810 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:48:34.640016 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:34.626135 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 31 19:48:34.767138 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:34.763964 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 19:48:34.781491 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:34.781451 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:34.781777 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:34.781743 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.126218 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.126175 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 31 19:48:35.155117 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.155053 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.186088 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.185951 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 19:48:35.243939 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.242995 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.314902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.308080 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.324420 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.324383 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:35.327992 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.327952 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.624448 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.624415 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 19:48:35.633124 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.631692 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 31 19:48:35.649902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.641991 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.649902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.649111 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 19:48:35.651060 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.650263 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 19:48:35.669057 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.667658 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:48:35.678087 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.676972 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:48:35.714912 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.711601 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:48:35.719156 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.719131 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:48:35.765330 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.765304 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:48:35.767817 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:35.767206 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:48:49.695525 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.695446 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl"] Oct 31 19:48:49.696212 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.696194 6107 cpu_manager.go:414] "containerMap: removing container" podUID="41cf9a87-976b-4d0c-a145-30ce809450d0" containerName="revoke-keys" Oct 31 19:48:49.696268 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.696214 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cf9a87-976b-4d0c-a145-30ce809450d0" containerName="revoke-keys" Oct 31 19:48:49.696339 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.696330 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cf9a87-976b-4d0c-a145-30ce809450d0" containerName="revoke-keys" Oct 31 19:48:49.701143 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.701124 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.757561 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.757536 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-31a1ba53-openshift-default-dockercfg-zzn8l\"" Oct 31 19:48:49.771457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.771432 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-token\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.771610 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.771591 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-certs\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.771671 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.771625 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-envoy\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.771774 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.771752 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-socket\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.771841 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.771783 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-credential-socket\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.771965 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.771944 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-data\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.772008 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.771986 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istiod-ca-cert\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.772089 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.772072 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v446\" (UniqueName: \"kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-kube-api-access-6v446\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.772148 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.772123 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-podinfo\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.802083 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.802057 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl"] Oct 31 19:48:49.873610 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.873572 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-certs\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.873773 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.873628 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-envoy\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.873773 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.873683 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-socket\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.873773 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.873698 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-credential-socket\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.873920 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.873773 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-data\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.873920 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.873799 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istiod-ca-cert\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.873920 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.873840 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v446\" (UniqueName: \"kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-kube-api-access-6v446\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.873920 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.873890 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-podinfo\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.874118 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.873942 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-token\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.874118 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.874010 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-certs\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.874219 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.874131 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-credential-socket\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.874219 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.874165 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-socket\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.874322 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.874231 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-data\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.874817 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.874793 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istiod-ca-cert\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.875869 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.875845 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-envoy\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.876464 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.876442 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-podinfo\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.934947 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.934858 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-token\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:49.935825 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:49.935792 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v446\" (UniqueName: \"kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-kube-api-access-6v446\") pod \"e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:50.015363 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:50.015332 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:50.406570 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:48:50.406445 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:48:50.486841 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:48:50.486815 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd838c9f4_2a42_4e0a_8ff2_2a385e677bd7.slice/crio-9d6ef0904250b92b49822d529a750f274ec73e3ae645ed39d7d73843aa85b1eb WatchSource:0}: Error finding container 9d6ef0904250b92b49822d529a750f274ec73e3ae645ed39d7d73843aa85b1eb: Status 404 returned error can't find the container with id 9d6ef0904250b92b49822d529a750f274ec73e3ae645ed39d7d73843aa85b1eb Oct 31 19:48:50.489334 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:50.489301 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:48:50.489436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:50.489377 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:48:50.489436 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:50.489421 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:48:50.489691 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:50.489669 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl"] Oct 31 19:48:51.074005 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:51.073971 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" event={"ID":"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7","Type":"ContainerStarted","Data":"b3c98dff6871c19ddd337a38545b428a5dc3366d812fdbc3d065f493c82f8d20"} Oct 31 19:48:51.074511 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:51.074024 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" event={"ID":"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7","Type":"ContainerStarted","Data":"9d6ef0904250b92b49822d529a750f274ec73e3ae645ed39d7d73843aa85b1eb"} Oct 31 19:48:51.185710 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:51.185648 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" podStartSLOduration=2.185632119 podStartE2EDuration="2.185632119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:51.183373133 +0000 UTC m=+1902.816767078" watchObservedRunningTime="2026-10-31 19:48:51.185632119 +0000 UTC m=+1902.819026026" Oct 31 19:48:52.015787 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:52.015747 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:52.021159 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:52.021133 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:52.080940 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:52.080914 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:52.082013 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:52.081991 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:48:55.079806 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:55.079767 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr"] Oct 31 19:48:55.089400 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:55.089375 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" Oct 31 19:48:55.138772 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:55.138745 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr"] Oct 31 19:48:55.194903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:55.183239 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ae778663\"" Oct 31 19:48:55.194903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:55.183380 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ae778663-dockercfg-l848s\"" Oct 31 19:48:55.307454 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:55.307414 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwqh\" (UniqueName: \"kubernetes.io/projected/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-kube-api-access-gtwqh\") pod \"payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr\" (UID: \"f59f774f-5c2d-47ca-b8c9-bf9e1f281228\") " pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" Oct 31 19:48:55.307772 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:55.307749 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr\" (UID: \"f59f774f-5c2d-47ca-b8c9-bf9e1f281228\") " pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" Oct 31 19:48:55.422669 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:55.422578 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwqh\" (UniqueName: \"kubernetes.io/projected/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-kube-api-access-gtwqh\") pod \"payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr\" (UID: \"f59f774f-5c2d-47ca-b8c9-bf9e1f281228\") " pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" Oct 31 19:48:55.423088 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:55.423034 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr\" (UID: \"f59f774f-5c2d-47ca-b8c9-bf9e1f281228\") " pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" Oct 31 19:48:55.424098 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:55.424052 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr\" (UID: \"f59f774f-5c2d-47ca-b8c9-bf9e1f281228\") " pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" Oct 31 19:48:55.475982 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:55.475948 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwqh\" (UniqueName: \"kubernetes.io/projected/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-kube-api-access-gtwqh\") pod \"payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr\" (UID: \"f59f774f-5c2d-47ca-b8c9-bf9e1f281228\") " pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" Oct 31 19:48:55.719125 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:55.719092 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" Oct 31 19:48:56.095475 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:56.095443 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr"] Oct 31 19:48:56.095955 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:48:56.095933 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59f774f_5c2d_47ca_b8c9_bf9e1f281228.slice/crio-213b34771191a9b2fe05bc657af176f15acd720b45f7adaebbcd80488fcdc553 WatchSource:0}: Error finding container 213b34771191a9b2fe05bc657af176f15acd720b45f7adaebbcd80488fcdc553: Status 404 returned error can't find the container with id 213b34771191a9b2fe05bc657af176f15acd720b45f7adaebbcd80488fcdc553 Oct 31 19:48:56.114047 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:56.114018 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" event={"ID":"f59f774f-5c2d-47ca-b8c9-bf9e1f281228","Type":"ContainerStarted","Data":"213b34771191a9b2fe05bc657af176f15acd720b45f7adaebbcd80488fcdc553"} Oct 31 19:48:57.123591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:57.123549 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" event={"ID":"f59f774f-5c2d-47ca-b8c9-bf9e1f281228","Type":"ContainerStarted","Data":"a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616"} Oct 31 19:48:57.124117 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:57.123617 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" Oct 31 19:48:57.234578 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:48:57.234527 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" podStartSLOduration=2.234511505 podStartE2EDuration="2.234511505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:57.232665103 +0000 UTC m=+1908.866059011" watchObservedRunningTime="2026-10-31 19:48:57.234511505 +0000 UTC m=+1908.867905413" Oct 31 19:49:00.200921 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:00.199953 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m"] Oct 31 19:49:00.237374 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:00.237349 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" Oct 31 19:49:00.262991 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:00.262940 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m"] Oct 31 19:49:00.342938 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:00.342576 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvrss\" (UniqueName: \"kubernetes.io/projected/15b2c0d0-5596-4189-bbaa-992f33f8f4df-kube-api-access-vvrss\") pod \"payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m\" (UID: \"15b2c0d0-5596-4189-bbaa-992f33f8f4df\") " pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" Oct 31 19:49:00.342938 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:00.342795 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b2c0d0-5596-4189-bbaa-992f33f8f4df-plugins-config-volume\") pod \"payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m\" (UID: \"15b2c0d0-5596-4189-bbaa-992f33f8f4df\") " pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" Oct 31 19:49:00.370364 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:00.370159 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-31a1ba53\"" Oct 31 19:49:00.370364 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:00.370281 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-31a1ba53-dockercfg-tftht\"" Oct 31 19:49:00.444445 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:00.444405 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvrss\" (UniqueName: \"kubernetes.io/projected/15b2c0d0-5596-4189-bbaa-992f33f8f4df-kube-api-access-vvrss\") pod \"payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m\" (UID: \"15b2c0d0-5596-4189-bbaa-992f33f8f4df\") " pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" Oct 31 19:49:00.444645 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:00.444632 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b2c0d0-5596-4189-bbaa-992f33f8f4df-plugins-config-volume\") pod \"payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m\" (UID: \"15b2c0d0-5596-4189-bbaa-992f33f8f4df\") " pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" Oct 31 19:49:00.445587 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:00.445553 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b2c0d0-5596-4189-bbaa-992f33f8f4df-plugins-config-volume\") pod \"payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m\" (UID: \"15b2c0d0-5596-4189-bbaa-992f33f8f4df\") " pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" Oct 31 19:49:00.497491 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:00.497454 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvrss\" (UniqueName: \"kubernetes.io/projected/15b2c0d0-5596-4189-bbaa-992f33f8f4df-kube-api-access-vvrss\") pod \"payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m\" (UID: \"15b2c0d0-5596-4189-bbaa-992f33f8f4df\") " pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" Oct 31 19:49:00.626548 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:00.626515 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" Oct 31 19:49:01.019964 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:49:01.019929 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b2c0d0_5596_4189_bbaa_992f33f8f4df.slice/crio-486ad7be24f340186a2dd7e60f1735a7e2a07767a74ac08bfe759a58118b9f28 WatchSource:0}: Error finding container 486ad7be24f340186a2dd7e60f1735a7e2a07767a74ac08bfe759a58118b9f28: Status 404 returned error can't find the container with id 486ad7be24f340186a2dd7e60f1735a7e2a07767a74ac08bfe759a58118b9f28 Oct 31 19:49:01.020785 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:01.020751 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m"] Oct 31 19:49:01.336914 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:01.336792 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" event={"ID":"15b2c0d0-5596-4189-bbaa-992f33f8f4df","Type":"ContainerStarted","Data":"7e490b2d8ff43a9714dfcd712e8dd9f9b77085285cf57f45d99a6acd99323091"} Oct 31 19:49:01.336914 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:01.336849 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" event={"ID":"15b2c0d0-5596-4189-bbaa-992f33f8f4df","Type":"ContainerStarted","Data":"486ad7be24f340186a2dd7e60f1735a7e2a07767a74ac08bfe759a58118b9f28"} Oct 31 19:49:01.336914 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:01.336877 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" Oct 31 19:49:01.444527 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:01.444478 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" podStartSLOduration=1.444465012 podStartE2EDuration="1.444465012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:01.441537352 +0000 UTC m=+1913.074931269" watchObservedRunningTime="2026-10-31 19:49:01.444465012 +0000 UTC m=+1913.077858919" Oct 31 19:49:08.135623 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:08.135592 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" Oct 31 19:49:12.349145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:12.349100 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" Oct 31 19:49:20.404439 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:49:20.404397 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:49:36.719677 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.719646 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7"] Oct 31 19:49:36.724341 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.724322 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:36.766859 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.766828 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-maas-api-tls\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:36.767017 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.766966 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5262fe98-69d3-40c8-bbed-afc712746098-openshift-service-ca\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:36.767078 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.767061 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp5gp\" (UniqueName: \"kubernetes.io/projected/5262fe98-69d3-40c8-bbed-afc712746098-kube-api-access-tp5gp\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:36.767290 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.767264 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-metrics-certs\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:36.773051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.772874 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ae778663-serving-cert\"" Oct 31 19:49:36.778581 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.778553 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7"] Oct 31 19:49:36.869314 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.869274 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp5gp\" (UniqueName: \"kubernetes.io/projected/5262fe98-69d3-40c8-bbed-afc712746098-kube-api-access-tp5gp\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:36.869479 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.869387 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-metrics-certs\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:36.869479 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.869452 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-maas-api-tls\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:36.869564 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.869530 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5262fe98-69d3-40c8-bbed-afc712746098-openshift-service-ca\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:36.869709 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:49:36.869681 6107 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-serving-cert: secret "maas-api-e2e-ait-ae778663-serving-cert" not found Oct 31 19:49:36.869859 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:49:36.869752 6107 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-maas-api-tls podName:5262fe98-69d3-40c8-bbed-afc712746098 nodeName:}" failed. No retries permitted until 2026-10-31 19:49:37.369732114 +0000 UTC m=+1949.003126008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-maas-api-tls") pod "maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" (UID: "5262fe98-69d3-40c8-bbed-afc712746098") : secret "maas-api-e2e-ait-ae778663-serving-cert" not found Oct 31 19:49:36.870341 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.870309 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5262fe98-69d3-40c8-bbed-afc712746098-openshift-service-ca\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:36.872222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.872196 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-metrics-certs\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:36.923943 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:36.923864 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp5gp\" (UniqueName: \"kubernetes.io/projected/5262fe98-69d3-40c8-bbed-afc712746098-kube-api-access-tp5gp\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:37.378401 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:37.378369 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-maas-api-tls\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:37.380895 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:37.380862 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-maas-api-tls\") pod \"maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:37.637793 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:37.637700 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:38.022656 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:38.022624 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7"] Oct 31 19:49:38.023111 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:49:38.022985 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5262fe98_69d3_40c8_bbed_afc712746098.slice/crio-564ab5aa3052a009a60f639a2a7fd4e3c2f703e7dc13fbb19077b1b0a34c574f WatchSource:0}: Error finding container 564ab5aa3052a009a60f639a2a7fd4e3c2f703e7dc13fbb19077b1b0a34c574f: Status 404 returned error can't find the container with id 564ab5aa3052a009a60f639a2a7fd4e3c2f703e7dc13fbb19077b1b0a34c574f Oct 31 19:49:38.727769 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:38.727733 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" event={"ID":"5262fe98-69d3-40c8-bbed-afc712746098","Type":"ContainerStarted","Data":"564ab5aa3052a009a60f639a2a7fd4e3c2f703e7dc13fbb19077b1b0a34c574f"} Oct 31 19:49:39.737868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:39.737829 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" event={"ID":"5262fe98-69d3-40c8-bbed-afc712746098","Type":"ContainerStarted","Data":"0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6"} Oct 31 19:49:39.738351 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:39.738033 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:39.850485 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:39.850433 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" podStartSLOduration=2.980353529 podStartE2EDuration="3.850417163s" totalImagesPullingTime="870.063634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:39.845184647 +0000 UTC m=+1951.478578555" watchObservedRunningTime="2026-10-31 19:49:39.850417163 +0000 UTC m=+1951.483811071" Oct 31 19:49:45.761929 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:45.761846 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:49:49.281125 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.281090 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d"] Oct 31 19:49:49.286166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.286144 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.347736 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.347687 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8zpx\" (UniqueName: \"kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-kube-api-access-d8zpx\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.347944 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.347737 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.347944 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.347817 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.347944 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.347915 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.348150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.347950 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.348150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.347996 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.348150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.348024 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.348310 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.348160 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.348310 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.348264 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.396996 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.396961 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d"] Oct 31 19:49:49.450135 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.450099 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.450333 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.450148 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.450333 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.450186 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.450333 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.450213 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.450535 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.450340 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.450535 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.450443 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.450656 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.450542 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8zpx\" (UniqueName: \"kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-kube-api-access-d8zpx\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.450656 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.450572 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.450656 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.450577 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.450812 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.450660 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.450871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.450812 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.451318 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.451268 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.451527 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.451500 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.451692 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.451584 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.454127 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.454086 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.454237 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.454146 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.504910 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.504847 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.505139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.505113 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8zpx\" (UniqueName: \"kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-kube-api-access-d8zpx\") pod \"maas-default-gateway-openshift-default-58d898bd55-frv2d\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:49.602410 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:49.602329 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:50.031310 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:50.031280 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d"] Oct 31 19:49:50.032309 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:49:50.032284 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709870d0_a4b9_4e4c_8c4b_52058d9e5e1c.slice/crio-2d04a66025985c533642e3b3e9f84965b2c56198511623769b2d3a948f1a096d WatchSource:0}: Error finding container 2d04a66025985c533642e3b3e9f84965b2c56198511623769b2d3a948f1a096d: Status 404 returned error can't find the container with id 2d04a66025985c533642e3b3e9f84965b2c56198511623769b2d3a948f1a096d Oct 31 19:49:50.035148 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:50.035114 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:49:50.035246 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:50.035193 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:49:50.035246 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:50.035241 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:49:50.415589 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:49:50.415497 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:49:50.987704 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:50.987658 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" event={"ID":"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c","Type":"ContainerStarted","Data":"fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba"} Oct 31 19:49:50.987932 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:50.987736 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" event={"ID":"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c","Type":"ContainerStarted","Data":"2d04a66025985c533642e3b3e9f84965b2c56198511623769b2d3a948f1a096d"} Oct 31 19:49:51.142635 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:51.142586 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" podStartSLOduration=2.142569933 podStartE2EDuration="2.142569933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:51.14110646 +0000 UTC m=+1962.774500378" watchObservedRunningTime="2026-10-31 19:49:51.142569933 +0000 UTC m=+1962.775963841" Oct 31 19:49:51.606905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:51.604166 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:51.613900 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:51.607541 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-frv2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:49:51.613900 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:51.607616 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 19:49:52.605278 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:52.605244 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-frv2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:49:52.609904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:52.605529 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 19:49:53.604320 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:53.604274 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-frv2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:49:53.604541 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:53.604337 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 19:49:54.607314 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:54.607275 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-frv2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:49:54.607921 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:54.607864 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 19:49:55.604309 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:55.604271 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-frv2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:49:55.605097 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:55.605057 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 19:49:56.603603 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:56.603570 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-frv2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:49:56.604116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:56.603627 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 19:49:57.603053 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:57.603012 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-frv2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:49:57.603231 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:57.603078 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 19:49:58.766596 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:58.766558 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:58.767081 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:58.766923 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:49:58.767804 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:49:58.767785 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:50:20.436333 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:50:20.436238 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:50:28.327855 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:28.327821 6107 generic.go:356] "Generic (PLEG): container finished" podID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerID="b3c98dff6871c19ddd337a38545b428a5dc3366d812fdbc3d065f493c82f8d20" exitCode=0 Oct 31 19:50:28.328949 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:28.327904 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" event={"ID":"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7","Type":"ContainerDied","Data":"b3c98dff6871c19ddd337a38545b428a5dc3366d812fdbc3d065f493c82f8d20"} Oct 31 19:50:28.328949 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:28.328450 6107 scope.go:122] "RemoveContainer" containerID="b3c98dff6871c19ddd337a38545b428a5dc3366d812fdbc3d065f493c82f8d20" Oct 31 19:50:28.330448 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:28.330413 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:50:28.330591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:28.330502 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:50:28.330591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:28.330558 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:50:29.359975 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:29.359938 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" event={"ID":"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7","Type":"ContainerStarted","Data":"75ec3c70f23f498075b524baa50b020130f54372c773b1f0b6f027e274fc5901"} Oct 31 19:50:30.022906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:30.020988 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:50:30.022906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:30.021147 6107 patch_prober.go:29] interesting pod/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:50:30.022906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:30.021192 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 19:50:31.023904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:31.023271 6107 patch_prober.go:29] interesting pod/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:50:31.023904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:31.023344 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 19:50:32.025453 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:32.025418 6107 patch_prober.go:29] interesting pod/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:50:32.026352 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:32.026309 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 19:50:33.025905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:33.018205 6107 patch_prober.go:29] interesting pod/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:50:33.025905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:33.018265 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 19:50:34.016904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:34.016241 6107 patch_prober.go:29] interesting pod/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:50:34.016904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:34.016305 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 19:50:35.021213 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:35.021176 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:50:35.021733 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:35.021711 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:50:35.022588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:35.022567 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:50:50.407839 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:50:50.407793 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:50:55.564568 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:50:55.564533 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr"] Oct 31 19:51:00.565704 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:00.565649 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" podUID="f59f774f-5c2d-47ca-b8c9-bf9e1f281228" containerName="payload-processing" containerID="cri-o://a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616" gracePeriod=25 Oct 31 19:51:00.870975 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:00.870949 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" Oct 31 19:51:00.918496 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:00.918462 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-kube-api-access-gtwqh\" (UniqueName: \"kubernetes.io/projected/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-kube-api-access-gtwqh\") pod \"f59f774f-5c2d-47ca-b8c9-bf9e1f281228\" (UID: \"f59f774f-5c2d-47ca-b8c9-bf9e1f281228\") " Oct 31 19:51:00.918704 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:00.918594 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-plugins-config-volume\") pod \"f59f774f-5c2d-47ca-b8c9-bf9e1f281228\" (UID: \"f59f774f-5c2d-47ca-b8c9-bf9e1f281228\") " Oct 31 19:51:00.919030 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:00.919001 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-plugins-config-volume" pod "f59f774f-5c2d-47ca-b8c9-bf9e1f281228" (UID: "f59f774f-5c2d-47ca-b8c9-bf9e1f281228"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:00.919457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:00.919433 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:00.921654 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:00.921627 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-kube-api-access-gtwqh" pod "f59f774f-5c2d-47ca-b8c9-bf9e1f281228" (UID: "f59f774f-5c2d-47ca-b8c9-bf9e1f281228"). InnerVolumeSpecName "kube-api-access-gtwqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:01.021254 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:01.021218 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtwqh\" (UniqueName: \"kubernetes.io/projected/f59f774f-5c2d-47ca-b8c9-bf9e1f281228-kube-api-access-gtwqh\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:01.638441 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:01.638401 6107 generic.go:356] "Generic (PLEG): container finished" podID="f59f774f-5c2d-47ca-b8c9-bf9e1f281228" containerID="a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616" exitCode=0 Oct 31 19:51:01.638852 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:01.638444 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" event={"ID":"f59f774f-5c2d-47ca-b8c9-bf9e1f281228","Type":"ContainerDied","Data":"a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616"} Oct 31 19:51:01.638852 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:01.638504 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" event={"ID":"f59f774f-5c2d-47ca-b8c9-bf9e1f281228","Type":"ContainerDied","Data":"213b34771191a9b2fe05bc657af176f15acd720b45f7adaebbcd80488fcdc553"} Oct 31 19:51:01.638852 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:01.638533 6107 scope.go:122] "RemoveContainer" containerID="a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616" Oct 31 19:51:01.638852 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:01.638456 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr" Oct 31 19:51:01.651356 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:01.651332 6107 scope.go:122] "RemoveContainer" containerID="a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616" Oct 31 19:51:01.651634 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:51:01.651614 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616\": container with ID starting with a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616 not found: ID does not exist" containerID="a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616" Oct 31 19:51:01.651694 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:01.651645 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616"} err="failed to get container status \"a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616\": rpc error: code = NotFound desc = could not find container \"a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616\": container with ID starting with a9af34caa8dc6a17704ae11802ed832380bbf797e62b4e8d56323b25c850d616 not found: ID does not exist" Oct 31 19:51:01.792008 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:01.791974 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr"] Oct 31 19:51:01.838188 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:01.838155 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae778663-d9c9cf9d5-g5wbr"] Oct 31 19:51:02.359167 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:02.359132 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m"] Oct 31 19:51:02.492998 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:02.492964 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59f774f-5c2d-47ca-b8c9-bf9e1f281228" path="/var/lib/kubelet/pods/f59f774f-5c2d-47ca-b8c9-bf9e1f281228/volumes" Oct 31 19:51:07.360904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:07.359967 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" podUID="15b2c0d0-5596-4189-bbaa-992f33f8f4df" containerName="payload-processing" containerID="cri-o://7e490b2d8ff43a9714dfcd712e8dd9f9b77085285cf57f45d99a6acd99323091" gracePeriod=25 Oct 31 19:51:07.739904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:07.739314 6107 generic.go:356] "Generic (PLEG): container finished" podID="15b2c0d0-5596-4189-bbaa-992f33f8f4df" containerID="7e490b2d8ff43a9714dfcd712e8dd9f9b77085285cf57f45d99a6acd99323091" exitCode=0 Oct 31 19:51:07.739904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:07.739505 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" event={"ID":"15b2c0d0-5596-4189-bbaa-992f33f8f4df","Type":"ContainerDied","Data":"7e490b2d8ff43a9714dfcd712e8dd9f9b77085285cf57f45d99a6acd99323091"} Oct 31 19:51:08.698968 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:08.694143 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:51:08.698968 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:08.696339 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" Oct 31 19:51:08.698968 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:08.697528 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b2c0d0-5596-4189-bbaa-992f33f8f4df-kube-api-access-vvrss\" (UniqueName: \"kubernetes.io/projected/15b2c0d0-5596-4189-bbaa-992f33f8f4df-kube-api-access-vvrss\") pod \"15b2c0d0-5596-4189-bbaa-992f33f8f4df\" (UID: \"15b2c0d0-5596-4189-bbaa-992f33f8f4df\") " Oct 31 19:51:08.698968 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:08.697771 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b2c0d0-5596-4189-bbaa-992f33f8f4df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b2c0d0-5596-4189-bbaa-992f33f8f4df-plugins-config-volume\") pod \"15b2c0d0-5596-4189-bbaa-992f33f8f4df\" (UID: \"15b2c0d0-5596-4189-bbaa-992f33f8f4df\") " Oct 31 19:51:08.705873 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:08.705805 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b2c0d0-5596-4189-bbaa-992f33f8f4df-plugins-config-volume" pod "15b2c0d0-5596-4189-bbaa-992f33f8f4df" (UID: "15b2c0d0-5596-4189-bbaa-992f33f8f4df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:08.733688 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:08.733633 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b2c0d0-5596-4189-bbaa-992f33f8f4df-kube-api-access-vvrss" pod "15b2c0d0-5596-4189-bbaa-992f33f8f4df" (UID: "15b2c0d0-5596-4189-bbaa-992f33f8f4df"). InnerVolumeSpecName "kube-api-access-vvrss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:08.836905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:08.810217 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b2c0d0-5596-4189-bbaa-992f33f8f4df-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:08.836905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:08.810259 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvrss\" (UniqueName: \"kubernetes.io/projected/15b2c0d0-5596-4189-bbaa-992f33f8f4df-kube-api-access-vvrss\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:09.000145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:09.000104 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" event={"ID":"15b2c0d0-5596-4189-bbaa-992f33f8f4df","Type":"ContainerDied","Data":"486ad7be24f340186a2dd7e60f1735a7e2a07767a74ac08bfe759a58118b9f28"} Oct 31 19:51:09.000408 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:09.000389 6107 scope.go:122] "RemoveContainer" containerID="7e490b2d8ff43a9714dfcd712e8dd9f9b77085285cf57f45d99a6acd99323091" Oct 31 19:51:09.000756 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:09.000736 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m" Oct 31 19:51:09.043559 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:51:09.043516 6107 kuberuntime_gc.go:392] "Failed to remove container log dead symlink" err="remove /var/log/containers/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m_openshift-ingress_payload-processing-7e490b2d8ff43a9714dfcd712e8dd9f9b77085285cf57f45d99a6acd99323091.log: no such file or directory" path="/var/log/containers/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m_openshift-ingress_payload-processing-7e490b2d8ff43a9714dfcd712e8dd9f9b77085285cf57f45d99a6acd99323091.log" Oct 31 19:51:09.087028 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:09.086976 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:51:09.263917 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:09.255876 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m"] Oct 31 19:51:09.264130 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:09.264107 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31a1ba53-68cb4c7789-ndn5m"] Oct 31 19:51:10.490767 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:10.490733 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b2c0d0-5596-4189-bbaa-992f33f8f4df" path="/var/lib/kubelet/pods/15b2c0d0-5596-4189-bbaa-992f33f8f4df/volumes" Oct 31 19:51:20.405699 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:51:20.405662 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:51:23.441674 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.441642 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx"] Oct 31 19:51:23.442317 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.442304 6107 cpu_manager.go:414] "containerMap: removing container" podUID="f59f774f-5c2d-47ca-b8c9-bf9e1f281228" containerName="payload-processing" Oct 31 19:51:23.442365 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.442319 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59f774f-5c2d-47ca-b8c9-bf9e1f281228" containerName="payload-processing" Oct 31 19:51:23.442365 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.442347 6107 cpu_manager.go:414] "containerMap: removing container" podUID="15b2c0d0-5596-4189-bbaa-992f33f8f4df" containerName="payload-processing" Oct 31 19:51:23.442365 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.442352 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b2c0d0-5596-4189-bbaa-992f33f8f4df" containerName="payload-processing" Oct 31 19:51:23.442469 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.442458 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59f774f-5c2d-47ca-b8c9-bf9e1f281228" containerName="payload-processing" Oct 31 19:51:23.442511 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.442474 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b2c0d0-5596-4189-bbaa-992f33f8f4df" containerName="payload-processing" Oct 31 19:51:23.447409 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.447391 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.558904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.558425 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-data\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.558904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.558479 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch25v\" (UniqueName: \"kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-kube-api-access-ch25v\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.558904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.558629 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-token\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.558904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.558703 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-certs\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.558904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.558749 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e05a2c-520b-46a6-80cb-60672de0f2c9-istiod-ca-cert\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.558904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.558873 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-socket\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.559365 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.558988 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-envoy\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.559365 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.559022 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-podinfo\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.559365 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.559080 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-credential-socket\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.598203 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.598170 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-abb7688a-gw-openshift-default-dockercfg-pb6z6\"" Oct 31 19:51:23.598749 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.598725 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx"] Oct 31 19:51:23.660689 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.660654 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-token\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.661008 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.660986 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-certs\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.661455 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.661192 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e05a2c-520b-46a6-80cb-60672de0f2c9-istiod-ca-cert\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.663937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.661784 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-socket\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.664135 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.664117 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-envoy\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.664257 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.664242 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-podinfo\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.664385 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.664373 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-credential-socket\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.664511 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.664496 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-data\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.664631 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.664601 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch25v\" (UniqueName: \"kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-kube-api-access-ch25v\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.665743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.665721 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-certs\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.666479 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.666463 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e05a2c-520b-46a6-80cb-60672de0f2c9-istiod-ca-cert\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.666748 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.666736 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-socket\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.673950 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.670177 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-credential-socket\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.691839 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.691760 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-podinfo\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.705997 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.692365 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-data\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.708326 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.708179 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-envoy\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.752763 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.752679 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch25v\" (UniqueName: \"kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-kube-api-access-ch25v\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.755233 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.755175 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-token\") pod \"e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:23.761420 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:23.761182 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:24.245645 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:24.245611 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx"] Oct 31 19:51:24.248016 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:51:24.247988 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e05a2c_520b_46a6_80cb_60672de0f2c9.slice/crio-9cecb29f2944c0e644880f734fee857c24d235415c502c48e86dad32ef269530 WatchSource:0}: Error finding container 9cecb29f2944c0e644880f734fee857c24d235415c502c48e86dad32ef269530: Status 404 returned error can't find the container with id 9cecb29f2944c0e644880f734fee857c24d235415c502c48e86dad32ef269530 Oct 31 19:51:24.251495 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:24.251454 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:51:24.251586 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:24.251544 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:51:24.251624 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:24.251587 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:51:25.170964 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:25.170923 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" event={"ID":"47e05a2c-520b-46a6-80cb-60672de0f2c9","Type":"ContainerStarted","Data":"a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3"} Oct 31 19:51:25.171435 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:25.170983 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" event={"ID":"47e05a2c-520b-46a6-80cb-60672de0f2c9","Type":"ContainerStarted","Data":"9cecb29f2944c0e644880f734fee857c24d235415c502c48e86dad32ef269530"} Oct 31 19:51:25.326983 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:25.326933 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" podStartSLOduration=2.326919153 podStartE2EDuration="2.326919153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:25.32442009 +0000 UTC m=+2056.957814000" watchObservedRunningTime="2026-10-31 19:51:25.326919153 +0000 UTC m=+2056.960313131" Oct 31 19:51:25.762554 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:25.762519 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:25.767950 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:25.767923 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:26.178073 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.177993 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:26.179024 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.179007 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:51:26.356908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.354993 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7"] Oct 31 19:51:26.356908 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.355243 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" podUID="5262fe98-69d3-40c8-bbed-afc712746098" containerName="maas-api" containerID="cri-o://0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6" gracePeriod=30 Oct 31 19:51:26.749960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.749603 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:51:26.855488 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.855452 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5262fe98-69d3-40c8-bbed-afc712746098-kube-api-access-tp5gp\" (UniqueName: \"kubernetes.io/projected/5262fe98-69d3-40c8-bbed-afc712746098-kube-api-access-tp5gp\") pod \"5262fe98-69d3-40c8-bbed-afc712746098\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " Oct 31 19:51:26.855762 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.855580 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5262fe98-69d3-40c8-bbed-afc712746098-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5262fe98-69d3-40c8-bbed-afc712746098-openshift-service-ca\") pod \"5262fe98-69d3-40c8-bbed-afc712746098\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " Oct 31 19:51:26.855860 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.855808 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-maas-api-tls\") pod \"5262fe98-69d3-40c8-bbed-afc712746098\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " Oct 31 19:51:26.856005 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.855928 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-metrics-certs\") pod \"5262fe98-69d3-40c8-bbed-afc712746098\" (UID: \"5262fe98-69d3-40c8-bbed-afc712746098\") " Oct 31 19:51:26.857645 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.857391 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5262fe98-69d3-40c8-bbed-afc712746098-openshift-service-ca" pod "5262fe98-69d3-40c8-bbed-afc712746098" (UID: "5262fe98-69d3-40c8-bbed-afc712746098"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:26.861491 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.861303 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5262fe98-69d3-40c8-bbed-afc712746098-kube-api-access-tp5gp" pod "5262fe98-69d3-40c8-bbed-afc712746098" (UID: "5262fe98-69d3-40c8-bbed-afc712746098"). InnerVolumeSpecName "kube-api-access-tp5gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:26.864003 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.863924 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-metrics-certs" pod "5262fe98-69d3-40c8-bbed-afc712746098" (UID: "5262fe98-69d3-40c8-bbed-afc712746098"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:26.864003 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.863963 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-maas-api-tls" pod "5262fe98-69d3-40c8-bbed-afc712746098" (UID: "5262fe98-69d3-40c8-bbed-afc712746098"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:26.958377 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.958344 6107 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5262fe98-69d3-40c8-bbed-afc712746098-openshift-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:26.958377 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.958374 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-maas-api-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:26.958588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.958387 6107 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5262fe98-69d3-40c8-bbed-afc712746098-metrics-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:26.958588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:26.958397 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp5gp\" (UniqueName: \"kubernetes.io/projected/5262fe98-69d3-40c8-bbed-afc712746098-kube-api-access-tp5gp\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.186299 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:27.186206 6107 generic.go:356] "Generic (PLEG): container finished" podID="5262fe98-69d3-40c8-bbed-afc712746098" containerID="0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6" exitCode=0 Oct 31 19:51:27.186299 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:27.186252 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" Oct 31 19:51:27.186807 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:27.186296 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" event={"ID":"5262fe98-69d3-40c8-bbed-afc712746098","Type":"ContainerDied","Data":"0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6"} Oct 31 19:51:27.186807 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:27.186359 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7" event={"ID":"5262fe98-69d3-40c8-bbed-afc712746098","Type":"ContainerDied","Data":"564ab5aa3052a009a60f639a2a7fd4e3c2f703e7dc13fbb19077b1b0a34c574f"} Oct 31 19:51:27.186807 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:27.186391 6107 scope.go:122] "RemoveContainer" containerID="0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6" Oct 31 19:51:27.198306 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:27.198285 6107 scope.go:122] "RemoveContainer" containerID="0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6" Oct 31 19:51:27.198586 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:51:27.198566 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6\": container with ID starting with 0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6 not found: ID does not exist" containerID="0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6" Oct 31 19:51:27.198644 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:27.198594 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6"} err="failed to get container status \"0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6\": rpc error: code = NotFound desc = could not find container \"0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6\": container with ID starting with 0786c2055555f1b818cc7d0ffd53934db947fe551f28fb8724459e8eb4931df6 not found: ID does not exist" Oct 31 19:51:27.384903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:27.359724 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7"] Oct 31 19:51:27.441960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:27.433147 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae778663-8548c5df6d-pc6l7"] Oct 31 19:51:28.489578 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:28.489535 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5262fe98-69d3-40c8-bbed-afc712746098" path="/var/lib/kubelet/pods/5262fe98-69d3-40c8-bbed-afc712746098/volumes" Oct 31 19:51:30.828905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:30.827963 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z"] Oct 31 19:51:30.829428 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:30.829145 6107 cpu_manager.go:414] "containerMap: removing container" podUID="5262fe98-69d3-40c8-bbed-afc712746098" containerName="maas-api" Oct 31 19:51:30.829428 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:30.829167 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5262fe98-69d3-40c8-bbed-afc712746098" containerName="maas-api" Oct 31 19:51:30.834936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:30.833290 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="5262fe98-69d3-40c8-bbed-afc712746098" containerName="maas-api" Oct 31 19:51:30.851906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:30.851190 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" Oct 31 19:51:30.877068 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:30.877033 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z"] Oct 31 19:51:30.941906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:30.930834 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-abb7688a-dockercfg-6pmdw\"" Oct 31 19:51:30.941906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:30.930998 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-abb7688a\"" Oct 31 19:51:31.016536 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.016496 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2mqp\" (UniqueName: \"kubernetes.io/projected/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-kube-api-access-c2mqp\") pod \"payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z\" (UID: \"aeb70bb7-9fd7-4bf6-919c-471c10996fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" Oct 31 19:51:31.016743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.016624 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z\" (UID: \"aeb70bb7-9fd7-4bf6-919c-471c10996fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" Oct 31 19:51:31.026249 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.022896 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn"] Oct 31 19:51:31.072364 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.072323 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn"] Oct 31 19:51:31.072611 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.072435 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" Oct 31 19:51:31.134203 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.121456 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2mqp\" (UniqueName: \"kubernetes.io/projected/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-kube-api-access-c2mqp\") pod \"payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z\" (UID: \"aeb70bb7-9fd7-4bf6-919c-471c10996fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" Oct 31 19:51:31.134203 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.121629 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z\" (UID: \"aeb70bb7-9fd7-4bf6-919c-471c10996fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" Oct 31 19:51:31.134203 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.131161 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z\" (UID: \"aeb70bb7-9fd7-4bf6-919c-471c10996fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" Oct 31 19:51:31.235851 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.233529 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znk2v\" (UniqueName: \"kubernetes.io/projected/184afc07-989d-48e1-a13e-ac34c03b4615-kube-api-access-znk2v\") pod \"payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn\" (UID: \"184afc07-989d-48e1-a13e-ac34c03b4615\") " pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" Oct 31 19:51:31.235851 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.233784 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184afc07-989d-48e1-a13e-ac34c03b4615-plugins-config-volume\") pod \"payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn\" (UID: \"184afc07-989d-48e1-a13e-ac34c03b4615\") " pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" Oct 31 19:51:31.272512 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.270663 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2mqp\" (UniqueName: \"kubernetes.io/projected/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-kube-api-access-c2mqp\") pod \"payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z\" (UID: \"aeb70bb7-9fd7-4bf6-919c-471c10996fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" Oct 31 19:51:31.335576 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.335525 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184afc07-989d-48e1-a13e-ac34c03b4615-plugins-config-volume\") pod \"payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn\" (UID: \"184afc07-989d-48e1-a13e-ac34c03b4615\") " pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" Oct 31 19:51:31.335916 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.335893 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk2v\" (UniqueName: \"kubernetes.io/projected/184afc07-989d-48e1-a13e-ac34c03b4615-kube-api-access-znk2v\") pod \"payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn\" (UID: \"184afc07-989d-48e1-a13e-ac34c03b4615\") " pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" Oct 31 19:51:31.336773 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.336744 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184afc07-989d-48e1-a13e-ac34c03b4615-plugins-config-volume\") pod \"payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn\" (UID: \"184afc07-989d-48e1-a13e-ac34c03b4615\") " pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" Oct 31 19:51:31.389548 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.389463 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znk2v\" (UniqueName: \"kubernetes.io/projected/184afc07-989d-48e1-a13e-ac34c03b4615-kube-api-access-znk2v\") pod \"payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn\" (UID: \"184afc07-989d-48e1-a13e-ac34c03b4615\") " pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" Oct 31 19:51:31.474932 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.473190 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" Oct 31 19:51:31.487151 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:31.485236 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" Oct 31 19:51:32.071965 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:51:32.071930 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb70bb7_9fd7_4bf6_919c_471c10996fbf.slice/crio-d998c8df0faa516cd09da3dbf9f6ffc25d00c5a93f9dfbff028f05503a363c83 WatchSource:0}: Error finding container d998c8df0faa516cd09da3dbf9f6ffc25d00c5a93f9dfbff028f05503a363c83: Status 404 returned error can't find the container with id d998c8df0faa516cd09da3dbf9f6ffc25d00c5a93f9dfbff028f05503a363c83 Oct 31 19:51:32.073333 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:32.073301 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z"] Oct 31 19:51:32.118708 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:32.118671 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn"] Oct 31 19:51:32.119178 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:51:32.119149 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184afc07_989d_48e1_a13e_ac34c03b4615.slice/crio-5be350bcbb6373b64c76aa4c3621b1a83303f887284de81525104e9b8b0e84b0 WatchSource:0}: Error finding container 5be350bcbb6373b64c76aa4c3621b1a83303f887284de81525104e9b8b0e84b0: Status 404 returned error can't find the container with id 5be350bcbb6373b64c76aa4c3621b1a83303f887284de81525104e9b8b0e84b0 Oct 31 19:51:32.256113 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:32.256060 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" event={"ID":"aeb70bb7-9fd7-4bf6-919c-471c10996fbf","Type":"ContainerStarted","Data":"31d40ec1c16b3363ee78068b1991d510afea488a98d74e4d09fb1cb422a4ae9e"} Oct 31 19:51:32.256301 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:32.256128 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" event={"ID":"aeb70bb7-9fd7-4bf6-919c-471c10996fbf","Type":"ContainerStarted","Data":"d998c8df0faa516cd09da3dbf9f6ffc25d00c5a93f9dfbff028f05503a363c83"} Oct 31 19:51:32.256301 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:32.256154 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" Oct 31 19:51:32.258023 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:32.257992 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" event={"ID":"184afc07-989d-48e1-a13e-ac34c03b4615","Type":"ContainerStarted","Data":"911d30dcf7dcb6b265cb5360deccca12f7c4f2c8578587b05e40ab71d2982d78"} Oct 31 19:51:32.258167 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:32.258043 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" event={"ID":"184afc07-989d-48e1-a13e-ac34c03b4615","Type":"ContainerStarted","Data":"5be350bcbb6373b64c76aa4c3621b1a83303f887284de81525104e9b8b0e84b0"} Oct 31 19:51:32.258167 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:32.258076 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" Oct 31 19:51:32.362636 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:32.362522 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" podStartSLOduration=2.362506286 podStartE2EDuration="2.362506286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:32.360908639 +0000 UTC m=+2063.994302547" watchObservedRunningTime="2026-10-31 19:51:32.362506286 +0000 UTC m=+2063.995900193" Oct 31 19:51:32.469508 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:32.469456 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" podStartSLOduration=2.469438131 podStartE2EDuration="2.469438131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:32.467176174 +0000 UTC m=+2064.100570082" watchObservedRunningTime="2026-10-31 19:51:32.469438131 +0000 UTC m=+2064.102832063" Oct 31 19:51:36.158997 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:36.158923 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl"] Oct 31 19:51:36.159672 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:36.159252 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerName="istio-proxy" containerID="cri-o://75ec3c70f23f498075b524baa50b020130f54372c773b1f0b6f027e274fc5901" gracePeriod=30 Oct 31 19:51:41.369960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.357678 6107 generic.go:356] "Generic (PLEG): container finished" podID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerID="75ec3c70f23f498075b524baa50b020130f54372c773b1f0b6f027e274fc5901" exitCode=0 Oct 31 19:51:41.369960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.357912 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" event={"ID":"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7","Type":"ContainerDied","Data":"75ec3c70f23f498075b524baa50b020130f54372c773b1f0b6f027e274fc5901"} Oct 31 19:51:41.369960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.357979 6107 scope.go:122] "RemoveContainer" containerID="b3c98dff6871c19ddd337a38545b428a5dc3366d812fdbc3d065f493c82f8d20" Oct 31 19:51:41.525774 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.525741 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf"] Oct 31 19:51:41.534391 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.534366 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" Oct 31 19:51:41.616949 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.592383 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:51:41.621347 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.621311 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf"] Oct 31 19:51:41.631144 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.631118 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-84fd8e82\"" Oct 31 19:51:41.631518 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.631469 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-84fd8e82-dockercfg-fdrz7\"" Oct 31 19:51:41.708899 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.705816 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw"] Oct 31 19:51:41.708899 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.707001 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerName="istio-proxy" Oct 31 19:51:41.708899 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.707021 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerName="istio-proxy" Oct 31 19:51:41.708899 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.707085 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerName="istio-proxy" Oct 31 19:51:41.708899 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.707093 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerName="istio-proxy" Oct 31 19:51:41.708899 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.707260 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerName="istio-proxy" Oct 31 19:51:41.712383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.710897 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-credential-socket\") pod \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " Oct 31 19:51:41.712383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.710956 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-envoy\") pod \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " Oct 31 19:51:41.712383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.711034 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istiod-ca-cert\") pod \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " Oct 31 19:51:41.712383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.711067 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-socket\") pod \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " Oct 31 19:51:41.712383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.711133 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-certs\") pod \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " Oct 31 19:51:41.712383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.711285 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-token\" (UniqueName: \"kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-token\") pod \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " Oct 31 19:51:41.712383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.711335 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-kube-api-access-6v446\" (UniqueName: \"kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-kube-api-access-6v446\") pod \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " Oct 31 19:51:41.712383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.711363 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-data\") pod \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " Oct 31 19:51:41.712383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.711414 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-podinfo\") pod \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\" (UID: \"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7\") " Oct 31 19:51:41.712383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.711812 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1ec68c-5951-4a09-bc5a-788fa51609df-plugins-config-volume\") pod \"payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf\" (UID: \"5f1ec68c-5951-4a09-bc5a-788fa51609df\") " pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" Oct 31 19:51:41.712383 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.711899 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntmtg\" (UniqueName: \"kubernetes.io/projected/5f1ec68c-5951-4a09-bc5a-788fa51609df-kube-api-access-ntmtg\") pod \"payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf\" (UID: \"5f1ec68c-5951-4a09-bc5a-788fa51609df\") " pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" Oct 31 19:51:41.713003 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.712908 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-credential-socket" pod "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" (UID: "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:41.714482 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.714047 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-data" pod "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" (UID: "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:41.714482 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.714281 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-socket" pod "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" (UID: "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:41.714482 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.714453 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-certs" pod "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" (UID: "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:41.715184 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.715163 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istiod-ca-cert" pod "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" (UID: "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:41.722788 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.722769 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" Oct 31 19:51:41.728970 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.728918 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-envoy" pod "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" (UID: "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:41.729325 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.729278 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-podinfo" pod "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" (UID: "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:51:41.729853 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.729814 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-token" pod "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" (UID: "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:41.730263 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.730233 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-kube-api-access-6v446" pod "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" (UID: "d838c9f4-2a42-4e0a-8ff2-2a385e677bd7"). InnerVolumeSpecName "kube-api-access-6v446". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:41.762554 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.762517 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw"] Oct 31 19:51:41.816479 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.816389 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrfn\" (UniqueName: \"kubernetes.io/projected/0a1857e4-b97e-420a-8a65-1bfa3277864f-kube-api-access-mdrfn\") pod \"payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw\" (UID: \"0a1857e4-b97e-420a-8a65-1bfa3277864f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" Oct 31 19:51:41.816854 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.816824 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1ec68c-5951-4a09-bc5a-788fa51609df-plugins-config-volume\") pod \"payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf\" (UID: \"5f1ec68c-5951-4a09-bc5a-788fa51609df\") " pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" Oct 31 19:51:41.817099 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.817079 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1857e4-b97e-420a-8a65-1bfa3277864f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw\" (UID: \"0a1857e4-b97e-420a-8a65-1bfa3277864f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" Oct 31 19:51:41.817239 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.817222 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntmtg\" (UniqueName: \"kubernetes.io/projected/5f1ec68c-5951-4a09-bc5a-788fa51609df-kube-api-access-ntmtg\") pod \"payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf\" (UID: \"5f1ec68c-5951-4a09-bc5a-788fa51609df\") " pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" Oct 31 19:51:41.817509 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.817492 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:41.818955 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.818938 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:41.819084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.819071 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:41.819185 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.819173 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:41.819291 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.819278 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:41.819384 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.819371 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:41.819469 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.819456 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v446\" (UniqueName: \"kubernetes.io/projected/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-kube-api-access-6v446\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:41.819554 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.819542 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:41.819643 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.819628 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:41.819724 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.818481 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1ec68c-5951-4a09-bc5a-788fa51609df-plugins-config-volume\") pod \"payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf\" (UID: \"5f1ec68c-5951-4a09-bc5a-788fa51609df\") " pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" Oct 31 19:51:41.872823 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.872787 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntmtg\" (UniqueName: \"kubernetes.io/projected/5f1ec68c-5951-4a09-bc5a-788fa51609df-kube-api-access-ntmtg\") pod \"payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf\" (UID: \"5f1ec68c-5951-4a09-bc5a-788fa51609df\") " pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" Oct 31 19:51:41.921136 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.921091 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrfn\" (UniqueName: \"kubernetes.io/projected/0a1857e4-b97e-420a-8a65-1bfa3277864f-kube-api-access-mdrfn\") pod \"payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw\" (UID: \"0a1857e4-b97e-420a-8a65-1bfa3277864f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" Oct 31 19:51:41.921318 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.921298 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1857e4-b97e-420a-8a65-1bfa3277864f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw\" (UID: \"0a1857e4-b97e-420a-8a65-1bfa3277864f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" Oct 31 19:51:41.922079 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.922056 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1857e4-b97e-420a-8a65-1bfa3277864f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw\" (UID: \"0a1857e4-b97e-420a-8a65-1bfa3277864f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" Oct 31 19:51:41.946460 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.946425 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" Oct 31 19:51:41.974175 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:41.974146 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrfn\" (UniqueName: \"kubernetes.io/projected/0a1857e4-b97e-420a-8a65-1bfa3277864f-kube-api-access-mdrfn\") pod \"payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw\" (UID: \"0a1857e4-b97e-420a-8a65-1bfa3277864f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" Oct 31 19:51:42.092302 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:42.092216 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" Oct 31 19:51:42.330394 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:51:42.330360 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1ec68c_5951_4a09_bc5a_788fa51609df.slice/crio-9ca2b990e6fc6e3c8e7b92f44dc9b998a00b404ca58614468abde1545310743d WatchSource:0}: Error finding container 9ca2b990e6fc6e3c8e7b92f44dc9b998a00b404ca58614468abde1545310743d: Status 404 returned error can't find the container with id 9ca2b990e6fc6e3c8e7b92f44dc9b998a00b404ca58614468abde1545310743d Oct 31 19:51:42.330689 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:42.330668 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf"] Oct 31 19:51:42.368541 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:42.368507 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" event={"ID":"5f1ec68c-5951-4a09-bc5a-788fa51609df","Type":"ContainerStarted","Data":"9ca2b990e6fc6e3c8e7b92f44dc9b998a00b404ca58614468abde1545310743d"} Oct 31 19:51:42.370113 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:42.370086 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" event={"ID":"d838c9f4-2a42-4e0a-8ff2-2a385e677bd7","Type":"ContainerDied","Data":"9d6ef0904250b92b49822d529a750f274ec73e3ae645ed39d7d73843aa85b1eb"} Oct 31 19:51:42.370494 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:42.370142 6107 scope.go:122] "RemoveContainer" containerID="75ec3c70f23f498075b524baa50b020130f54372c773b1f0b6f027e274fc5901" Oct 31 19:51:42.370494 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:42.370148 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl" Oct 31 19:51:42.490251 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:51:42.490218 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1857e4_b97e_420a_8a65_1bfa3277864f.slice/crio-87b7fb2d8c14ebf0ec9cdba6bf2a65c388fe945a6dc855d483dbe0d4abb1ab59 WatchSource:0}: Error finding container 87b7fb2d8c14ebf0ec9cdba6bf2a65c388fe945a6dc855d483dbe0d4abb1ab59: Status 404 returned error can't find the container with id 87b7fb2d8c14ebf0ec9cdba6bf2a65c388fe945a6dc855d483dbe0d4abb1ab59 Oct 31 19:51:42.491801 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:42.491776 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw"] Oct 31 19:51:42.525624 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:42.525580 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl"] Oct 31 19:51:42.530370 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:42.530342 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-31a1ba53-openshift-default-d46df6869-mllxl"] Oct 31 19:51:43.274364 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:43.274332 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" Oct 31 19:51:43.274538 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:43.274409 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" Oct 31 19:51:43.380537 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:43.380492 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" event={"ID":"5f1ec68c-5951-4a09-bc5a-788fa51609df","Type":"ContainerStarted","Data":"fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c"} Oct 31 19:51:43.381158 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:43.381140 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" Oct 31 19:51:43.384267 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:43.384234 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" event={"ID":"0a1857e4-b97e-420a-8a65-1bfa3277864f","Type":"ContainerStarted","Data":"0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be"} Oct 31 19:51:43.384400 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:43.384281 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" event={"ID":"0a1857e4-b97e-420a-8a65-1bfa3277864f","Type":"ContainerStarted","Data":"87b7fb2d8c14ebf0ec9cdba6bf2a65c388fe945a6dc855d483dbe0d4abb1ab59"} Oct 31 19:51:43.385098 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:43.385081 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" Oct 31 19:51:43.595997 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:43.595877 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" podStartSLOduration=2.595859644 podStartE2EDuration="2.595859644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:43.594001746 +0000 UTC m=+2075.227395656" watchObservedRunningTime="2026-10-31 19:51:43.595859644 +0000 UTC m=+2075.229253573" Oct 31 19:51:43.704218 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:43.704160 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" podStartSLOduration=2.704144593 podStartE2EDuration="2.704144593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:43.700553889 +0000 UTC m=+2075.333947798" watchObservedRunningTime="2026-10-31 19:51:43.704144593 +0000 UTC m=+2075.337538500" Oct 31 19:51:44.488297 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:44.488260 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" path="/var/lib/kubelet/pods/d838c9f4-2a42-4e0a-8ff2-2a385e677bd7/volumes" Oct 31 19:51:50.404814 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:51:50.404729 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:51:54.394636 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:54.394603 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" Oct 31 19:51:55.401270 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:55.401240 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" Oct 31 19:51:57.327410 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.327378 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9"] Oct 31 19:51:57.328326 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.328307 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d838c9f4-2a42-4e0a-8ff2-2a385e677bd7" containerName="istio-proxy" Oct 31 19:51:57.333400 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.333378 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.389926 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.389852 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9"] Oct 31 19:51:57.427953 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.427915 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 19:51:57.427953 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.427932 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 19:51:57.428180 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.427933 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 19:51:57.428180 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.427970 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hnzhb\"" Oct 31 19:51:57.474461 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.474430 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.474627 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.474517 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.474627 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.474569 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.474714 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.474689 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.474751 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.474741 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8b9x\" (UniqueName: \"kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-kube-api-access-c8b9x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.474790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.474771 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.474824 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.474799 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.576041 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.576001 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.576251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.576167 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.576251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.576232 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.576375 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.576280 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.576375 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.576315 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.576375 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.576348 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8b9x\" (UniqueName: \"kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-kube-api-access-c8b9x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.576538 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.576382 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.576871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.576845 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.578786 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.578721 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.578916 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.578895 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.579273 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.579251 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.579314 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.579281 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.629373 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.629339 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.629549 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.629519 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8b9x\" (UniqueName: \"kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-kube-api-access-c8b9x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7sfw9\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:57.647567 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:57.647536 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:51:58.075177 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:58.075131 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9"] Oct 31 19:51:58.110615 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:51:58.110584 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a00f61_6ddd_462c_a08a_7bd2eb1b226e.slice/crio-59b3f8b46b6d2568a799d92feb3b1a6b0786319bfb0705684226d1286899d3ef WatchSource:0}: Error finding container 59b3f8b46b6d2568a799d92feb3b1a6b0786319bfb0705684226d1286899d3ef: Status 404 returned error can't find the container with id 59b3f8b46b6d2568a799d92feb3b1a6b0786319bfb0705684226d1286899d3ef Oct 31 19:51:58.112602 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:58.112583 6107 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:51:58.507851 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:51:58.507817 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" event={"ID":"73a00f61-6ddd-462c-a08a-7bd2eb1b226e","Type":"ContainerStarted","Data":"59b3f8b46b6d2568a799d92feb3b1a6b0786319bfb0705684226d1286899d3ef"} Oct 31 19:52:02.477222 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:02.477189 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:52:02.477518 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:02.477263 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:52:03.561420 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:03.561383 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" event={"ID":"73a00f61-6ddd-462c-a08a-7bd2eb1b226e","Type":"ContainerStarted","Data":"8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a"} Oct 31 19:52:03.561829 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:03.561531 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:52:03.563175 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:03.563151 6107 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7sfw9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:52:03.563310 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:03.563192 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" podUID="73a00f61-6ddd-462c-a08a-7bd2eb1b226e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:52:03.759324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:03.759262 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" podStartSLOduration=2.396063846 podStartE2EDuration="6.759237469s" totalImagesPullingTime="4.363173623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:03.7234261 +0000 UTC m=+2095.356820009" watchObservedRunningTime="2026-10-31 19:52:03.759237469 +0000 UTC m=+2095.392631376" Oct 31 19:52:04.569677 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:04.569646 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 19:52:20.411730 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:52:20.411688 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:52:21.393917 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:21.393855 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897"] Oct 31 19:52:21.394108 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:21.394078 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" podUID="616e72de-9485-4db6-96e9-bfc30b9f639a" containerName="istio-proxy" containerID="cri-o://61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086" gracePeriod=30 Oct 31 19:52:26.136671 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.136630 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:52:26.137152 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.136718 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" podUID="616e72de-9485-4db6-96e9-bfc30b9f639a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:52:26.689922 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.689894 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:52:26.747011 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.746968 6107 generic.go:356] "Generic (PLEG): container finished" podID="616e72de-9485-4db6-96e9-bfc30b9f639a" containerID="61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086" exitCode=0 Oct 31 19:52:26.747211 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.747030 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" Oct 31 19:52:26.747211 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.747054 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" event={"ID":"616e72de-9485-4db6-96e9-bfc30b9f639a","Type":"ContainerDied","Data":"61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086"} Oct 31 19:52:26.747211 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.747124 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897" event={"ID":"616e72de-9485-4db6-96e9-bfc30b9f639a","Type":"ContainerDied","Data":"f29bb99b9309a9088edf44fcbd671a8bce71522024958cc22fa2973ae08f9eea"} Oct 31 19:52:26.747211 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.747153 6107 scope.go:122] "RemoveContainer" containerID="61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086" Oct 31 19:52:26.760929 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.760891 6107 scope.go:122] "RemoveContainer" containerID="61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086" Oct 31 19:52:26.761339 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:52:26.761311 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086\": container with ID starting with 61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086 not found: ID does not exist" containerID="61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086" Oct 31 19:52:26.761431 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.761350 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086"} err="failed to get container status \"61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086\": rpc error: code = NotFound desc = could not find container \"61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086\": container with ID starting with 61bae62d8896e5417a7420a6504db16dc5ac03e9717d5190545697e22f3ab086 not found: ID does not exist" Oct 31 19:52:26.787227 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.787195 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-certs\") pod \"616e72de-9485-4db6-96e9-bfc30b9f639a\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " Oct 31 19:52:26.787417 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.787252 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-token\" (UniqueName: \"kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-token\") pod \"616e72de-9485-4db6-96e9-bfc30b9f639a\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " Oct 31 19:52:26.787465 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.787417 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616e72de-9485-4db6-96e9-bfc30b9f639a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616e72de-9485-4db6-96e9-bfc30b9f639a-istiod-ca-cert\") pod \"616e72de-9485-4db6-96e9-bfc30b9f639a\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " Oct 31 19:52:26.787516 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.787492 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-socket\") pod \"616e72de-9485-4db6-96e9-bfc30b9f639a\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " Oct 31 19:52:26.787516 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.787507 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-certs" pod "616e72de-9485-4db6-96e9-bfc30b9f639a" (UID: "616e72de-9485-4db6-96e9-bfc30b9f639a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:26.787713 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.787691 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-credential-socket\") pod \"616e72de-9485-4db6-96e9-bfc30b9f639a\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " Oct 31 19:52:26.787924 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.787905 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-podinfo\") pod \"616e72de-9485-4db6-96e9-bfc30b9f639a\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " Oct 31 19:52:26.788056 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.788035 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-kube-api-access-z7bhj\" (UniqueName: \"kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-kube-api-access-z7bhj\") pod \"616e72de-9485-4db6-96e9-bfc30b9f639a\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " Oct 31 19:52:26.788156 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.788073 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-envoy\") pod \"616e72de-9485-4db6-96e9-bfc30b9f639a\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " Oct 31 19:52:26.788156 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.787802 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-socket" pod "616e72de-9485-4db6-96e9-bfc30b9f639a" (UID: "616e72de-9485-4db6-96e9-bfc30b9f639a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:26.788156 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.787895 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-credential-socket" pod "616e72de-9485-4db6-96e9-bfc30b9f639a" (UID: "616e72de-9485-4db6-96e9-bfc30b9f639a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:26.788156 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.787909 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616e72de-9485-4db6-96e9-bfc30b9f639a-istiod-ca-cert" pod "616e72de-9485-4db6-96e9-bfc30b9f639a" (UID: "616e72de-9485-4db6-96e9-bfc30b9f639a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:52:26.788156 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.788135 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-data\") pod \"616e72de-9485-4db6-96e9-bfc30b9f639a\" (UID: \"616e72de-9485-4db6-96e9-bfc30b9f639a\") " Oct 31 19:52:26.789139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.788762 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-data" pod "616e72de-9485-4db6-96e9-bfc30b9f639a" (UID: "616e72de-9485-4db6-96e9-bfc30b9f639a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:26.789139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.789047 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.789139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.789072 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616e72de-9485-4db6-96e9-bfc30b9f639a-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.789139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.789088 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.789139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.789102 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.789139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.789116 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.790234 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.790210 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-token" pod "616e72de-9485-4db6-96e9-bfc30b9f639a" (UID: "616e72de-9485-4db6-96e9-bfc30b9f639a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:52:26.790376 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.790355 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-podinfo" pod "616e72de-9485-4db6-96e9-bfc30b9f639a" (UID: "616e72de-9485-4db6-96e9-bfc30b9f639a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:52:26.790439 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.790372 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-envoy" pod "616e72de-9485-4db6-96e9-bfc30b9f639a" (UID: "616e72de-9485-4db6-96e9-bfc30b9f639a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:26.790543 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.790526 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-kube-api-access-z7bhj" pod "616e72de-9485-4db6-96e9-bfc30b9f639a" (UID: "616e72de-9485-4db6-96e9-bfc30b9f639a"). InnerVolumeSpecName "kube-api-access-z7bhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:52:26.890537 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.890496 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.890537 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.890541 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.890794 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.890557 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7bhj\" (UniqueName: \"kubernetes.io/projected/616e72de-9485-4db6-96e9-bfc30b9f639a-kube-api-access-z7bhj\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.890794 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:26.890572 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616e72de-9485-4db6-96e9-bfc30b9f639a-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:27.174721 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:27.174681 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897"] Oct 31 19:52:27.189683 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:27.189653 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-w6897"] Oct 31 19:52:28.488975 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:28.488940 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616e72de-9485-4db6-96e9-bfc30b9f639a" path="/var/lib/kubelet/pods/616e72de-9485-4db6-96e9-bfc30b9f639a/volumes" Oct 31 19:52:43.239907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:43.222023 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:43.342917 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:43.342855 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 19:52:43.420076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:43.420025 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:43.730182 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:43.730142 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:43.809829 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:43.809776 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 19:52:43.813080 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:43.813046 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 19:52:43.954380 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:43.954325 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:43.974066 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:43.974025 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:44.346906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.345975 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:44.346906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.346590 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:52:44.424032 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.423988 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" podUID="5019f17d-138f-4f54-b963-51645b530ffe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:44.427023 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.424829 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:52:44.790911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.784279 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:44.821909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.802514 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v" Oct 31 19:52:44.821909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.802584 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:52:44.821909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.810200 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:44.853049 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.853007 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:44.860365 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.860333 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:52:44.860646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.860629 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 19:52:44.860786 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.860774 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:52:44.879282 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.878938 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-frv2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:52:44.879282 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.879007 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:44.900903 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.896747 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 19:52:44.959129 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.958294 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:44.959129 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.958677 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:52:44.960111 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.960078 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 19:52:44.968425 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.964630 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 19:52:44.991438 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:44.991385 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 19:52:50.411180 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:52:50.411140 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:52:53.298589 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:53.298551 6107 generic.go:356] "Generic (PLEG): container finished" podID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerID="a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3" exitCode=0 Oct 31 19:52:53.299114 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:53.298623 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" event={"ID":"47e05a2c-520b-46a6-80cb-60672de0f2c9","Type":"ContainerDied","Data":"a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3"} Oct 31 19:52:53.299187 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:53.299171 6107 scope.go:122] "RemoveContainer" containerID="a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3" Oct 31 19:52:53.301102 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:53.301073 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:52:53.301232 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:53.301153 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:52:53.301232 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:53.301202 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:52:54.345034 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:54.344995 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" event={"ID":"47e05a2c-520b-46a6-80cb-60672de0f2c9","Type":"ContainerStarted","Data":"dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909"} Oct 31 19:52:54.762229 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:54.762194 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:52:54.763339 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:54.763314 6107 patch_prober.go:29] interesting pod/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 19:52:54.763457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:54.763370 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 19:52:55.762578 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:55.762541 6107 patch_prober.go:29] interesting pod/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 19:52:55.762993 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:55.762598 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 19:52:56.179045 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:56.178959 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:52:56.762147 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:56.762113 6107 patch_prober.go:29] interesting pod/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 19:52:56.762357 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:56.762164 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 19:52:56.949603 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:56.949553 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n"] Oct 31 19:52:56.950377 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:56.950358 6107 cpu_manager.go:414] "containerMap: removing container" podUID="616e72de-9485-4db6-96e9-bfc30b9f639a" containerName="istio-proxy" Oct 31 19:52:56.950377 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:56.950377 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616e72de-9485-4db6-96e9-bfc30b9f639a" containerName="istio-proxy" Oct 31 19:52:56.950533 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:56.950486 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="616e72de-9485-4db6-96e9-bfc30b9f639a" containerName="istio-proxy" Oct 31 19:52:56.954548 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:56.954510 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" Oct 31 19:52:56.993224 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:56.993189 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n"] Oct 31 19:52:57.037962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.033942 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-872fee\"" Oct 31 19:52:57.037962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.034127 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-872fee-dockercfg-mxf2b\"" Oct 31 19:52:57.081459 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.081419 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb4ld\" (UniqueName: \"kubernetes.io/projected/702ddccb-4d46-487e-ba61-1e40f396893f-kube-api-access-zb4ld\") pod \"payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n\" (UID: \"702ddccb-4d46-487e-ba61-1e40f396893f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" Oct 31 19:52:57.081948 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.081925 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702ddccb-4d46-487e-ba61-1e40f396893f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n\" (UID: \"702ddccb-4d46-487e-ba61-1e40f396893f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" Oct 31 19:52:57.149412 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.149361 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt"] Oct 31 19:52:57.168716 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.165146 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" Oct 31 19:52:57.184905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.184409 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702ddccb-4d46-487e-ba61-1e40f396893f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n\" (UID: \"702ddccb-4d46-487e-ba61-1e40f396893f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" Oct 31 19:52:57.197946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.186864 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt"] Oct 31 19:52:57.197946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.188243 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb4ld\" (UniqueName: \"kubernetes.io/projected/702ddccb-4d46-487e-ba61-1e40f396893f-kube-api-access-zb4ld\") pod \"payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n\" (UID: \"702ddccb-4d46-487e-ba61-1e40f396893f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" Oct 31 19:52:57.197946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.189258 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702ddccb-4d46-487e-ba61-1e40f396893f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n\" (UID: \"702ddccb-4d46-487e-ba61-1e40f396893f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" Oct 31 19:52:57.268308 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.266907 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb4ld\" (UniqueName: \"kubernetes.io/projected/702ddccb-4d46-487e-ba61-1e40f396893f-kube-api-access-zb4ld\") pod \"payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n\" (UID: \"702ddccb-4d46-487e-ba61-1e40f396893f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" Oct 31 19:52:57.309961 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.290156 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" Oct 31 19:52:57.309961 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.292159 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft7sh\" (UniqueName: \"kubernetes.io/projected/b0829bac-192a-4311-b773-2896d3b96490-kube-api-access-ft7sh\") pod \"payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt\" (UID: \"b0829bac-192a-4311-b773-2896d3b96490\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" Oct 31 19:52:57.309961 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.292265 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0829bac-192a-4311-b773-2896d3b96490-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt\" (UID: \"b0829bac-192a-4311-b773-2896d3b96490\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" Oct 31 19:52:57.393747 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.393669 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7sh\" (UniqueName: \"kubernetes.io/projected/b0829bac-192a-4311-b773-2896d3b96490-kube-api-access-ft7sh\") pod \"payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt\" (UID: \"b0829bac-192a-4311-b773-2896d3b96490\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" Oct 31 19:52:57.393747 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.393746 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0829bac-192a-4311-b773-2896d3b96490-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt\" (UID: \"b0829bac-192a-4311-b773-2896d3b96490\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" Oct 31 19:52:57.394533 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.394507 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0829bac-192a-4311-b773-2896d3b96490-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt\" (UID: \"b0829bac-192a-4311-b773-2896d3b96490\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" Oct 31 19:52:57.451465 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.451111 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft7sh\" (UniqueName: \"kubernetes.io/projected/b0829bac-192a-4311-b773-2896d3b96490-kube-api-access-ft7sh\") pod \"payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt\" (UID: \"b0829bac-192a-4311-b773-2896d3b96490\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" Oct 31 19:52:57.572960 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.569203 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" Oct 31 19:52:57.768770 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.768734 6107 patch_prober.go:29] interesting pod/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 19:52:57.768988 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.768795 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 19:52:57.795748 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:52:57.795720 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702ddccb_4d46_487e_ba61_1e40f396893f.slice/crio-383cd46cbdd44459bee80052ef7103bdf834895212c06673cf4d42f359233cea WatchSource:0}: Error finding container 383cd46cbdd44459bee80052ef7103bdf834895212c06673cf4d42f359233cea: Status 404 returned error can't find the container with id 383cd46cbdd44459bee80052ef7103bdf834895212c06673cf4d42f359233cea Oct 31 19:52:57.836931 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:57.836584 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n"] Oct 31 19:52:58.125796 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:58.125595 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt"] Oct 31 19:52:58.127841 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:52:58.127796 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0829bac_192a_4311_b773_2896d3b96490.slice/crio-1f4dabb88844384e1dd3aa23af99dfb7108fad7f6a6aa3066abc8bfe7eb97b1c WatchSource:0}: Error finding container 1f4dabb88844384e1dd3aa23af99dfb7108fad7f6a6aa3066abc8bfe7eb97b1c: Status 404 returned error can't find the container with id 1f4dabb88844384e1dd3aa23af99dfb7108fad7f6a6aa3066abc8bfe7eb97b1c Oct 31 19:52:58.384054 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:58.383957 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" event={"ID":"b0829bac-192a-4311-b773-2896d3b96490","Type":"ContainerStarted","Data":"aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a"} Oct 31 19:52:58.384054 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:58.384021 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" event={"ID":"b0829bac-192a-4311-b773-2896d3b96490","Type":"ContainerStarted","Data":"1f4dabb88844384e1dd3aa23af99dfb7108fad7f6a6aa3066abc8bfe7eb97b1c"} Oct 31 19:52:58.384054 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:58.384044 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" Oct 31 19:52:58.385803 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:58.385780 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" event={"ID":"702ddccb-4d46-487e-ba61-1e40f396893f","Type":"ContainerStarted","Data":"a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74"} Oct 31 19:52:58.386390 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:58.385817 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" event={"ID":"702ddccb-4d46-487e-ba61-1e40f396893f","Type":"ContainerStarted","Data":"383cd46cbdd44459bee80052ef7103bdf834895212c06673cf4d42f359233cea"} Oct 31 19:52:58.386390 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:58.385836 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" Oct 31 19:52:58.491146 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:58.491089 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" podStartSLOduration=1.4910755070000001 podStartE2EDuration="1.491075507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:58.489610155 +0000 UTC m=+2150.123004065" watchObservedRunningTime="2026-10-31 19:52:58.491075507 +0000 UTC m=+2150.124469415" Oct 31 19:52:58.651376 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:58.651249 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" podStartSLOduration=2.651226483 podStartE2EDuration="2.651226483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:58.600324558 +0000 UTC m=+2150.233718468" watchObservedRunningTime="2026-10-31 19:52:58.651226483 +0000 UTC m=+2150.284620392" Oct 31 19:52:58.762466 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:58.762430 6107 patch_prober.go:29] interesting pod/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 19:52:58.762660 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:58.762513 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 19:52:59.762656 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:59.762623 6107 patch_prober.go:29] interesting pod/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 19:52:59.763072 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:52:59.762675 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 19:53:00.767470 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:00.767442 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:53:00.768871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:00.768849 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:53:09.396720 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:09.396639 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" Oct 31 19:53:09.396720 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:09.396705 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" Oct 31 19:53:11.053129 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:11.053066 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v"] Oct 31 19:53:11.098743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:11.098702 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4563c9bf-29r6v"] Oct 31 19:53:12.488527 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:12.488492 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41cf9a87-976b-4d0c-a145-30ce809450d0" path="/var/lib/kubelet/pods/41cf9a87-976b-4d0c-a145-30ce809450d0/volumes" Oct 31 19:53:20.406088 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:53:20.405997 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:53:27.347862 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:27.347826 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn"] Oct 31 19:53:27.463028 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:27.462995 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z"] Oct 31 19:53:32.348404 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.348340 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" podUID="184afc07-989d-48e1-a13e-ac34c03b4615" containerName="payload-processing" containerID="cri-o://911d30dcf7dcb6b265cb5360deccca12f7c4f2c8578587b05e40ab71d2982d78" gracePeriod=25 Oct 31 19:53:32.464358 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.464285 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" podUID="aeb70bb7-9fd7-4bf6-919c-471c10996fbf" containerName="payload-pre-processing" containerID="cri-o://31d40ec1c16b3363ee78068b1991d510afea488a98d74e4d09fb1cb422a4ae9e" gracePeriod=25 Oct 31 19:53:32.709595 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.709258 6107 generic.go:356] "Generic (PLEG): container finished" podID="aeb70bb7-9fd7-4bf6-919c-471c10996fbf" containerID="31d40ec1c16b3363ee78068b1991d510afea488a98d74e4d09fb1cb422a4ae9e" exitCode=0 Oct 31 19:53:32.709595 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.709488 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" event={"ID":"aeb70bb7-9fd7-4bf6-919c-471c10996fbf","Type":"ContainerDied","Data":"31d40ec1c16b3363ee78068b1991d510afea488a98d74e4d09fb1cb422a4ae9e"} Oct 31 19:53:32.711734 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.711705 6107 generic.go:356] "Generic (PLEG): container finished" podID="184afc07-989d-48e1-a13e-ac34c03b4615" containerID="911d30dcf7dcb6b265cb5360deccca12f7c4f2c8578587b05e40ab71d2982d78" exitCode=0 Oct 31 19:53:32.712095 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.711944 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" event={"ID":"184afc07-989d-48e1-a13e-ac34c03b4615","Type":"ContainerDied","Data":"911d30dcf7dcb6b265cb5360deccca12f7c4f2c8578587b05e40ab71d2982d78"} Oct 31 19:53:32.712095 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.711996 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" event={"ID":"184afc07-989d-48e1-a13e-ac34c03b4615","Type":"ContainerDied","Data":"5be350bcbb6373b64c76aa4c3621b1a83303f887284de81525104e9b8b0e84b0"} Oct 31 19:53:32.712095 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.712022 6107 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5be350bcbb6373b64c76aa4c3621b1a83303f887284de81525104e9b8b0e84b0" Oct 31 19:53:32.713202 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.713173 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" Oct 31 19:53:32.769830 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.769800 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" Oct 31 19:53:32.806474 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.806437 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184afc07-989d-48e1-a13e-ac34c03b4615-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184afc07-989d-48e1-a13e-ac34c03b4615-plugins-config-volume\") pod \"184afc07-989d-48e1-a13e-ac34c03b4615\" (UID: \"184afc07-989d-48e1-a13e-ac34c03b4615\") " Oct 31 19:53:32.806925 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.806857 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184afc07-989d-48e1-a13e-ac34c03b4615-plugins-config-volume" pod "184afc07-989d-48e1-a13e-ac34c03b4615" (UID: "184afc07-989d-48e1-a13e-ac34c03b4615"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:53:32.807100 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.807074 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184afc07-989d-48e1-a13e-ac34c03b4615-kube-api-access-znk2v\" (UniqueName: \"kubernetes.io/projected/184afc07-989d-48e1-a13e-ac34c03b4615-kube-api-access-znk2v\") pod \"184afc07-989d-48e1-a13e-ac34c03b4615\" (UID: \"184afc07-989d-48e1-a13e-ac34c03b4615\") " Oct 31 19:53:32.807734 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.807712 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184afc07-989d-48e1-a13e-ac34c03b4615-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:32.809246 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.809224 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184afc07-989d-48e1-a13e-ac34c03b4615-kube-api-access-znk2v" pod "184afc07-989d-48e1-a13e-ac34c03b4615" (UID: "184afc07-989d-48e1-a13e-ac34c03b4615"). InnerVolumeSpecName "kube-api-access-znk2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:32.909379 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.909284 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-kube-api-access-c2mqp\" (UniqueName: \"kubernetes.io/projected/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-kube-api-access-c2mqp\") pod \"aeb70bb7-9fd7-4bf6-919c-471c10996fbf\" (UID: \"aeb70bb7-9fd7-4bf6-919c-471c10996fbf\") " Oct 31 19:53:32.909569 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.909433 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-plugins-config-volume\") pod \"aeb70bb7-9fd7-4bf6-919c-471c10996fbf\" (UID: \"aeb70bb7-9fd7-4bf6-919c-471c10996fbf\") " Oct 31 19:53:32.909804 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.909780 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-plugins-config-volume" pod "aeb70bb7-9fd7-4bf6-919c-471c10996fbf" (UID: "aeb70bb7-9fd7-4bf6-919c-471c10996fbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:53:32.910163 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.910148 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znk2v\" (UniqueName: \"kubernetes.io/projected/184afc07-989d-48e1-a13e-ac34c03b4615-kube-api-access-znk2v\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:32.910213 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.910168 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:32.911393 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:32.911371 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-kube-api-access-c2mqp" pod "aeb70bb7-9fd7-4bf6-919c-471c10996fbf" (UID: "aeb70bb7-9fd7-4bf6-919c-471c10996fbf"). InnerVolumeSpecName "kube-api-access-c2mqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:33.011789 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:33.011751 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2mqp\" (UniqueName: \"kubernetes.io/projected/aeb70bb7-9fd7-4bf6-919c-471c10996fbf-kube-api-access-c2mqp\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:33.723481 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:33.723452 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" Oct 31 19:53:33.724018 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:33.723496 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z" event={"ID":"aeb70bb7-9fd7-4bf6-919c-471c10996fbf","Type":"ContainerDied","Data":"d998c8df0faa516cd09da3dbf9f6ffc25d00c5a93f9dfbff028f05503a363c83"} Oct 31 19:53:33.724018 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:33.723565 6107 scope.go:122] "RemoveContainer" containerID="31d40ec1c16b3363ee78068b1991d510afea488a98d74e4d09fb1cb422a4ae9e" Oct 31 19:53:33.724018 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:33.723657 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn" Oct 31 19:53:33.924195 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:33.924161 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn"] Oct 31 19:53:33.970139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:33.970108 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-abb7688a-8459545575-f6zpn"] Oct 31 19:53:34.081570 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:34.081480 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z"] Oct 31 19:53:34.095296 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:34.095264 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-abb7688a-c7ddd9d7f-ck77z"] Oct 31 19:53:34.488934 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:34.488899 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184afc07-989d-48e1-a13e-ac34c03b4615" path="/var/lib/kubelet/pods/184afc07-989d-48e1-a13e-ac34c03b4615/volumes" Oct 31 19:53:34.489518 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:34.489500 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb70bb7-9fd7-4bf6-919c-471c10996fbf" path="/var/lib/kubelet/pods/aeb70bb7-9fd7-4bf6-919c-471c10996fbf/volumes" Oct 31 19:53:38.310266 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:38.310223 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf"] Oct 31 19:53:38.432838 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:38.432796 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw"] Oct 31 19:53:43.311385 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:43.311338 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" podUID="5f1ec68c-5951-4a09-bc5a-788fa51609df" containerName="payload-processing" containerID="cri-o://fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c" gracePeriod=25 Oct 31 19:53:43.434048 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:43.433995 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" podUID="0a1857e4-b97e-420a-8a65-1bfa3277864f" containerName="payload-pre-processing" containerID="cri-o://0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be" gracePeriod=25 Oct 31 19:53:43.614470 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:43.614437 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" Oct 31 19:53:43.761868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:43.750912 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f1ec68c-5951-4a09-bc5a-788fa51609df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1ec68c-5951-4a09-bc5a-788fa51609df-plugins-config-volume\") pod \"5f1ec68c-5951-4a09-bc5a-788fa51609df\" (UID: \"5f1ec68c-5951-4a09-bc5a-788fa51609df\") " Oct 31 19:53:43.761868 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:43.751044 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1ec68c-5951-4a09-bc5a-788fa51609df-kube-api-access-ntmtg\" (UniqueName: \"kubernetes.io/projected/5f1ec68c-5951-4a09-bc5a-788fa51609df-kube-api-access-ntmtg\") pod \"5f1ec68c-5951-4a09-bc5a-788fa51609df\" (UID: \"5f1ec68c-5951-4a09-bc5a-788fa51609df\") " Oct 31 19:53:43.767206 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:43.766898 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f1ec68c-5951-4a09-bc5a-788fa51609df-plugins-config-volume" pod "5f1ec68c-5951-4a09-bc5a-788fa51609df" (UID: "5f1ec68c-5951-4a09-bc5a-788fa51609df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:53:43.768139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:43.768102 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f1ec68c-5951-4a09-bc5a-788fa51609df-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:43.780549 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:43.780492 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1ec68c-5951-4a09-bc5a-788fa51609df-kube-api-access-ntmtg" pod "5f1ec68c-5951-4a09-bc5a-788fa51609df" (UID: "5f1ec68c-5951-4a09-bc5a-788fa51609df"). InnerVolumeSpecName "kube-api-access-ntmtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:43.874961 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:43.874691 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntmtg\" (UniqueName: \"kubernetes.io/projected/5f1ec68c-5951-4a09-bc5a-788fa51609df-kube-api-access-ntmtg\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:43.997967 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:43.989824 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" Oct 31 19:53:44.017219 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.017188 6107 generic.go:356] "Generic (PLEG): container finished" podID="5f1ec68c-5951-4a09-bc5a-788fa51609df" containerID="fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c" exitCode=0 Oct 31 19:53:44.017402 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.017240 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" Oct 31 19:53:44.017402 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.017267 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" event={"ID":"5f1ec68c-5951-4a09-bc5a-788fa51609df","Type":"ContainerDied","Data":"fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c"} Oct 31 19:53:44.017402 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.017323 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf" event={"ID":"5f1ec68c-5951-4a09-bc5a-788fa51609df","Type":"ContainerDied","Data":"9ca2b990e6fc6e3c8e7b92f44dc9b998a00b404ca58614468abde1545310743d"} Oct 31 19:53:44.017402 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.017346 6107 scope.go:122] "RemoveContainer" containerID="fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c" Oct 31 19:53:44.018982 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.018959 6107 generic.go:356] "Generic (PLEG): container finished" podID="0a1857e4-b97e-420a-8a65-1bfa3277864f" containerID="0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be" exitCode=0 Oct 31 19:53:44.019102 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.019002 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" Oct 31 19:53:44.019102 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.019046 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" event={"ID":"0a1857e4-b97e-420a-8a65-1bfa3277864f","Type":"ContainerDied","Data":"0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be"} Oct 31 19:53:44.019102 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.019091 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw" event={"ID":"0a1857e4-b97e-420a-8a65-1bfa3277864f","Type":"ContainerDied","Data":"87b7fb2d8c14ebf0ec9cdba6bf2a65c388fe945a6dc855d483dbe0d4abb1ab59"} Oct 31 19:53:44.030397 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.030372 6107 scope.go:122] "RemoveContainer" containerID="fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c" Oct 31 19:53:44.030735 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:53:44.030715 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c\": container with ID starting with fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c not found: ID does not exist" containerID="fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c" Oct 31 19:53:44.030810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.030748 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c"} err="failed to get container status \"fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c\": rpc error: code = NotFound desc = could not find container \"fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c\": container with ID starting with fce900a008b49070e168e134e2c927d83b906a98de9e9590a4532bd506a74d6c not found: ID does not exist" Oct 31 19:53:44.030810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.030773 6107 scope.go:122] "RemoveContainer" containerID="0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be" Oct 31 19:53:44.041405 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.041387 6107 scope.go:122] "RemoveContainer" containerID="0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be" Oct 31 19:53:44.041767 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:53:44.041746 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be\": container with ID starting with 0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be not found: ID does not exist" containerID="0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be" Oct 31 19:53:44.041838 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.041776 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be"} err="failed to get container status \"0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be\": rpc error: code = NotFound desc = could not find container \"0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be\": container with ID starting with 0f40573c3f623481001c423c0d4d8bb6d9f6fc5b1f044ff432e7ab788afd74be not found: ID does not exist" Oct 31 19:53:44.084450 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.084416 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1857e4-b97e-420a-8a65-1bfa3277864f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1857e4-b97e-420a-8a65-1bfa3277864f-plugins-config-volume\") pod \"0a1857e4-b97e-420a-8a65-1bfa3277864f\" (UID: \"0a1857e4-b97e-420a-8a65-1bfa3277864f\") " Oct 31 19:53:44.084646 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.084592 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1857e4-b97e-420a-8a65-1bfa3277864f-kube-api-access-mdrfn\" (UniqueName: \"kubernetes.io/projected/0a1857e4-b97e-420a-8a65-1bfa3277864f-kube-api-access-mdrfn\") pod \"0a1857e4-b97e-420a-8a65-1bfa3277864f\" (UID: \"0a1857e4-b97e-420a-8a65-1bfa3277864f\") " Oct 31 19:53:44.084840 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.084817 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1857e4-b97e-420a-8a65-1bfa3277864f-plugins-config-volume" pod "0a1857e4-b97e-420a-8a65-1bfa3277864f" (UID: "0a1857e4-b97e-420a-8a65-1bfa3277864f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:53:44.085343 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.085319 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1857e4-b97e-420a-8a65-1bfa3277864f-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:44.086944 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.086905 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1857e4-b97e-420a-8a65-1bfa3277864f-kube-api-access-mdrfn" pod "0a1857e4-b97e-420a-8a65-1bfa3277864f" (UID: "0a1857e4-b97e-420a-8a65-1bfa3277864f"). InnerVolumeSpecName "kube-api-access-mdrfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:44.192268 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.190594 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdrfn\" (UniqueName: \"kubernetes.io/projected/0a1857e4-b97e-420a-8a65-1bfa3277864f-kube-api-access-mdrfn\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:44.279539 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.279500 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf"] Oct 31 19:53:44.285974 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.285937 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-84fd8e82-6db68cfc7-56mdf"] Oct 31 19:53:44.446381 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.446290 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw"] Oct 31 19:53:44.453836 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.453801 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-84fd8e82-58d5f8496c-f6wjw"] Oct 31 19:53:44.488563 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.488533 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1857e4-b97e-420a-8a65-1bfa3277864f" path="/var/lib/kubelet/pods/0a1857e4-b97e-420a-8a65-1bfa3277864f/volumes" Oct 31 19:53:44.489051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:44.489036 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f1ec68c-5951-4a09-bc5a-788fa51609df" path="/var/lib/kubelet/pods/5f1ec68c-5951-4a09-bc5a-788fa51609df/volumes" Oct 31 19:53:50.017082 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.017025 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx"] Oct 31 19:53:50.018209 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.018185 6107 cpu_manager.go:414] "containerMap: removing container" podUID="5f1ec68c-5951-4a09-bc5a-788fa51609df" containerName="payload-processing" Oct 31 19:53:50.018322 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.018211 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1ec68c-5951-4a09-bc5a-788fa51609df" containerName="payload-processing" Oct 31 19:53:50.018322 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.018235 6107 cpu_manager.go:414] "containerMap: removing container" podUID="aeb70bb7-9fd7-4bf6-919c-471c10996fbf" containerName="payload-pre-processing" Oct 31 19:53:50.018322 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.018243 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb70bb7-9fd7-4bf6-919c-471c10996fbf" containerName="payload-pre-processing" Oct 31 19:53:50.018322 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.018295 6107 cpu_manager.go:414] "containerMap: removing container" podUID="0a1857e4-b97e-420a-8a65-1bfa3277864f" containerName="payload-pre-processing" Oct 31 19:53:50.018322 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.018303 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1857e4-b97e-420a-8a65-1bfa3277864f" containerName="payload-pre-processing" Oct 31 19:53:50.018500 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.018334 6107 cpu_manager.go:414] "containerMap: removing container" podUID="184afc07-989d-48e1-a13e-ac34c03b4615" containerName="payload-processing" Oct 31 19:53:50.018500 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.018342 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184afc07-989d-48e1-a13e-ac34c03b4615" containerName="payload-processing" Oct 31 19:53:50.018500 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.018493 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1857e4-b97e-420a-8a65-1bfa3277864f" containerName="payload-pre-processing" Oct 31 19:53:50.018593 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.018509 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="184afc07-989d-48e1-a13e-ac34c03b4615" containerName="payload-processing" Oct 31 19:53:50.018593 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.018524 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f1ec68c-5951-4a09-bc5a-788fa51609df" containerName="payload-processing" Oct 31 19:53:50.018593 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.018536 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb70bb7-9fd7-4bf6-919c-471c10996fbf" containerName="payload-pre-processing" Oct 31 19:53:50.024560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.024536 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" Oct 31 19:53:50.078639 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.074624 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx"] Oct 31 19:53:50.143912 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.137209 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9902ffd9\"" Oct 31 19:53:50.143912 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.137260 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9902ffd9-dockercfg-cfhnm\"" Oct 31 19:53:50.181706 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.167546 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4dw\" (UniqueName: \"kubernetes.io/projected/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-kube-api-access-vn4dw\") pod \"payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx\" (UID: \"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" Oct 31 19:53:50.181706 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.167757 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx\" (UID: \"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" Oct 31 19:53:50.197200 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.187119 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht"] Oct 31 19:53:50.280751 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.280618 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4dw\" (UniqueName: \"kubernetes.io/projected/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-kube-api-access-vn4dw\") pod \"payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx\" (UID: \"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" Oct 31 19:53:50.281012 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.280803 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx\" (UID: \"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" Oct 31 19:53:50.281992 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.281871 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx\" (UID: \"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" Oct 31 19:53:50.288237 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.288209 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" Oct 31 19:53:50.291507 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.291473 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht"] Oct 31 19:53:50.359491 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.359454 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4dw\" (UniqueName: \"kubernetes.io/projected/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-kube-api-access-vn4dw\") pod \"payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx\" (UID: \"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" Oct 31 19:53:50.408154 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:53:50.408110 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:53:50.484251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.484211 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tw2z\" (UniqueName: \"kubernetes.io/projected/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-kube-api-access-2tw2z\") pod \"payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht\" (UID: \"eff631f0-dbf7-4aeb-a148-69c5f7d38c54\") " pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" Oct 31 19:53:50.484435 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.484366 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-plugins-config-volume\") pod \"payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht\" (UID: \"eff631f0-dbf7-4aeb-a148-69c5f7d38c54\") " pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" Oct 31 19:53:50.585618 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.585515 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tw2z\" (UniqueName: \"kubernetes.io/projected/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-kube-api-access-2tw2z\") pod \"payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht\" (UID: \"eff631f0-dbf7-4aeb-a148-69c5f7d38c54\") " pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" Oct 31 19:53:50.585618 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.585610 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-plugins-config-volume\") pod \"payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht\" (UID: \"eff631f0-dbf7-4aeb-a148-69c5f7d38c54\") " pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" Oct 31 19:53:50.586414 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.586395 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-plugins-config-volume\") pod \"payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht\" (UID: \"eff631f0-dbf7-4aeb-a148-69c5f7d38c54\") " pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" Oct 31 19:53:50.650959 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.642349 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" Oct 31 19:53:50.650959 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.648490 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tw2z\" (UniqueName: \"kubernetes.io/projected/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-kube-api-access-2tw2z\") pod \"payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht\" (UID: \"eff631f0-dbf7-4aeb-a148-69c5f7d38c54\") " pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" Oct 31 19:53:50.915183 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:50.915081 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" Oct 31 19:53:51.062774 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:51.062737 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx"] Oct 31 19:53:51.068189 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:53:51.068163 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5ba5d4_7dfd_47da_ae42_649d38b8fa57.slice/crio-5e99328c26354bb1b51ceffacc79b3397a7c3fb850c5c70f1f0316e4492d8d64 WatchSource:0}: Error finding container 5e99328c26354bb1b51ceffacc79b3397a7c3fb850c5c70f1f0316e4492d8d64: Status 404 returned error can't find the container with id 5e99328c26354bb1b51ceffacc79b3397a7c3fb850c5c70f1f0316e4492d8d64 Oct 31 19:53:51.099974 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:51.099939 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" event={"ID":"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57","Type":"ContainerStarted","Data":"5e99328c26354bb1b51ceffacc79b3397a7c3fb850c5c70f1f0316e4492d8d64"} Oct 31 19:53:51.306557 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:53:51.306521 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff631f0_dbf7_4aeb_a148_69c5f7d38c54.slice/crio-0dd8739c1cb14cf19781030a4728bf07087dc713aed2ba7bc02ae80f95f17a0c WatchSource:0}: Error finding container 0dd8739c1cb14cf19781030a4728bf07087dc713aed2ba7bc02ae80f95f17a0c: Status 404 returned error can't find the container with id 0dd8739c1cb14cf19781030a4728bf07087dc713aed2ba7bc02ae80f95f17a0c Oct 31 19:53:51.307012 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:51.306982 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht"] Oct 31 19:53:52.122104 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:52.122063 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" event={"ID":"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57","Type":"ContainerStarted","Data":"620e42af4a5fe61764edf10599b34aa47e706f7624a1d966e53819e2863326fd"} Oct 31 19:53:52.122557 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:52.122291 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" Oct 31 19:53:52.124271 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:52.124244 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" event={"ID":"eff631f0-dbf7-4aeb-a148-69c5f7d38c54","Type":"ContainerStarted","Data":"b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93"} Oct 31 19:53:52.124399 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:52.124293 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" event={"ID":"eff631f0-dbf7-4aeb-a148-69c5f7d38c54","Type":"ContainerStarted","Data":"0dd8739c1cb14cf19781030a4728bf07087dc713aed2ba7bc02ae80f95f17a0c"} Oct 31 19:53:52.124399 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:52.124335 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" Oct 31 19:53:52.231986 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:52.231920 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" podStartSLOduration=3.231900102 podStartE2EDuration="3.231900102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:52.227845392 +0000 UTC m=+2203.861239299" watchObservedRunningTime="2026-10-31 19:53:52.231900102 +0000 UTC m=+2203.865294011" Oct 31 19:53:52.334752 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:53:52.334691 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" podStartSLOduration=2.334676312 podStartE2EDuration="2.334676312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:52.333076086 +0000 UTC m=+2203.966469997" watchObservedRunningTime="2026-10-31 19:53:52.334676312 +0000 UTC m=+2203.968070223" Oct 31 19:54:03.136978 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:03.136948 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" Oct 31 19:54:03.137458 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:03.137017 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" Oct 31 19:54:06.640446 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:06.640406 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj"] Oct 31 19:54:06.640986 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:06.640622 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" containerID="cri-o://bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0" gracePeriod=30 Oct 31 19:54:06.686422 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:06.686380 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch"] Oct 31 19:54:06.686703 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:06.686651 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerName="istio-proxy" containerID="cri-o://501b362e65d3d431d0d1dabc4d3d819fae7fb9b1bf8cc3892d638f3ef1a3178f" gracePeriod=30 Oct 31 19:54:06.690505 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:06.690478 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh"] Oct 31 19:54:06.690722 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:06.690699 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" podUID="b8691586-d76a-495b-9515-99f46a3e7040" containerName="istio-proxy" containerID="cri-o://1e55fd5efa74f3359af678cb8db55a476c6162f41590098bf94d6e0948db9134" gracePeriod=30 Oct 31 19:54:06.734055 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:06.734022 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh"] Oct 31 19:54:06.734289 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:06.734264 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" containerID="cri-o://0dd0ee4ea2f520ec9e9ba8c60af8032a473ad06920744115659595ce36d58d30" gracePeriod=30 Oct 31 19:54:09.097116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:09.097083 6107 scope.go:122] "RemoveContainer" containerID="e93eb6e6deb4fbb55ba757f9b60f4ae4df13fb23fa22c22ee69a64fc9202c757" Oct 31 19:54:11.811613 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:11.811578 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx"] Oct 31 19:54:11.812433 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:11.812402 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" containerID="cri-o://dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909" gracePeriod=30 Oct 31 19:54:12.025056 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.019768 6107 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:54:12.025056 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.019932 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" podUID="b8691586-d76a-495b-9515-99f46a3e7040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 19:54:12.198971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.198154 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:54:12.262708 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.262679 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-credential-socket\") pod \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " Oct 31 19:54:12.262915 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.262745 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-podinfo\") pod \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " Oct 31 19:54:12.262915 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.262773 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-envoy\") pod \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " Oct 31 19:54:12.262915 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.262853 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istiod-ca-cert\") pod \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " Oct 31 19:54:12.263182 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.262967 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-socket\") pod \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " Oct 31 19:54:12.263182 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.262996 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-kube-api-access-qc7pp\" (UniqueName: \"kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-kube-api-access-qc7pp\") pod \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " Oct 31 19:54:12.263295 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.263184 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-certs\") pod \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " Oct 31 19:54:12.263295 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.263272 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-token\" (UniqueName: \"kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-token\") pod \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " Oct 31 19:54:12.264112 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.263489 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-socket" pod "f9a7ba14-180b-4ee3-9917-e13736fd6ad2" (UID: "f9a7ba14-180b-4ee3-9917-e13736fd6ad2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.264112 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.264098 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-data\") pod \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\" (UID: \"f9a7ba14-180b-4ee3-9917-e13736fd6ad2\") " Oct 31 19:54:12.264301 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.263681 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-certs" pod "f9a7ba14-180b-4ee3-9917-e13736fd6ad2" (UID: "f9a7ba14-180b-4ee3-9917-e13736fd6ad2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.264301 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.263719 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istiod-ca-cert" pod "f9a7ba14-180b-4ee3-9917-e13736fd6ad2" (UID: "f9a7ba14-180b-4ee3-9917-e13736fd6ad2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:12.264301 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.263792 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-credential-socket" pod "f9a7ba14-180b-4ee3-9917-e13736fd6ad2" (UID: "f9a7ba14-180b-4ee3-9917-e13736fd6ad2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.264471 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.264415 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-data" pod "f9a7ba14-180b-4ee3-9917-e13736fd6ad2" (UID: "f9a7ba14-180b-4ee3-9917-e13736fd6ad2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.265052 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.265028 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.265189 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.265058 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.265189 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.265074 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.265316 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.265238 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.265316 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.265261 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.266910 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.266632 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-envoy" pod "f9a7ba14-180b-4ee3-9917-e13736fd6ad2" (UID: "f9a7ba14-180b-4ee3-9917-e13736fd6ad2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.266910 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.266734 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-podinfo" pod "f9a7ba14-180b-4ee3-9917-e13736fd6ad2" (UID: "f9a7ba14-180b-4ee3-9917-e13736fd6ad2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:12.270562 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.270530 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-kube-api-access-qc7pp" pod "f9a7ba14-180b-4ee3-9917-e13736fd6ad2" (UID: "f9a7ba14-180b-4ee3-9917-e13736fd6ad2"). InnerVolumeSpecName "kube-api-access-qc7pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:12.273016 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.272992 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-token" pod "f9a7ba14-180b-4ee3-9917-e13736fd6ad2" (UID: "f9a7ba14-180b-4ee3-9917-e13736fd6ad2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:12.347927 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.347858 6107 generic.go:356] "Generic (PLEG): container finished" podID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerID="501b362e65d3d431d0d1dabc4d3d819fae7fb9b1bf8cc3892d638f3ef1a3178f" exitCode=0 Oct 31 19:54:12.348164 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.348143 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" event={"ID":"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68","Type":"ContainerDied","Data":"501b362e65d3d431d0d1dabc4d3d819fae7fb9b1bf8cc3892d638f3ef1a3178f"} Oct 31 19:54:12.350262 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.348242 6107 scope.go:122] "RemoveContainer" containerID="6b5cfb6793e485acdd53250e965f475f85045545ab2291f3ddb26d367a169bbb" Oct 31 19:54:12.353081 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.353045 6107 generic.go:356] "Generic (PLEG): container finished" podID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerID="0dd0ee4ea2f520ec9e9ba8c60af8032a473ad06920744115659595ce36d58d30" exitCode=0 Oct 31 19:54:12.353207 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.353124 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" event={"ID":"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e","Type":"ContainerDied","Data":"0dd0ee4ea2f520ec9e9ba8c60af8032a473ad06920744115659595ce36d58d30"} Oct 31 19:54:12.354843 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.354816 6107 generic.go:356] "Generic (PLEG): container finished" podID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerID="bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0" exitCode=0 Oct 31 19:54:12.354973 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.354891 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" Oct 31 19:54:12.354973 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.354906 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" event={"ID":"f9a7ba14-180b-4ee3-9917-e13736fd6ad2","Type":"ContainerDied","Data":"bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0"} Oct 31 19:54:12.354973 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.354962 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj" event={"ID":"f9a7ba14-180b-4ee3-9917-e13736fd6ad2","Type":"ContainerDied","Data":"8adc1e71ba160a15c691334053f47b9405e8367a17eb6b8c8ad0482222f9e2d1"} Oct 31 19:54:12.355455 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.355432 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:54:12.359468 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.359446 6107 generic.go:356] "Generic (PLEG): container finished" podID="b8691586-d76a-495b-9515-99f46a3e7040" containerID="1e55fd5efa74f3359af678cb8db55a476c6162f41590098bf94d6e0948db9134" exitCode=0 Oct 31 19:54:12.359683 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.359655 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" event={"ID":"b8691586-d76a-495b-9515-99f46a3e7040","Type":"ContainerDied","Data":"1e55fd5efa74f3359af678cb8db55a476c6162f41590098bf94d6e0948db9134"} Oct 31 19:54:12.366544 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.366374 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.366544 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.366399 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.366544 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.366410 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.366544 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.366419 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc7pp\" (UniqueName: \"kubernetes.io/projected/f9a7ba14-180b-4ee3-9917-e13736fd6ad2-kube-api-access-qc7pp\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.373502 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.373478 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:54:12.374613 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.374586 6107 scope.go:122] "RemoveContainer" containerID="bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0" Oct 31 19:54:12.387518 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.387494 6107 scope.go:122] "RemoveContainer" containerID="bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0" Oct 31 19:54:12.387690 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.387652 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:54:12.387823 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:54:12.387784 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0\": container with ID starting with bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0 not found: ID does not exist" containerID="bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0" Oct 31 19:54:12.387867 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.387831 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0"} err="failed to get container status \"bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0\": rpc error: code = NotFound desc = could not find container \"bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0\": container with ID starting with bb769386e886ddffc96150339f476dc557150199b221b6eb212b9b1bca1fbea0 not found: ID does not exist" Oct 31 19:54:12.467173 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467138 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-data\") pod \"b8691586-d76a-495b-9515-99f46a3e7040\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " Oct 31 19:54:12.467359 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467183 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-podinfo\") pod \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " Oct 31 19:54:12.467359 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467251 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-envoy\") pod \"b8691586-d76a-495b-9515-99f46a3e7040\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " Oct 31 19:54:12.467359 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467300 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-socket\") pod \"b8691586-d76a-495b-9515-99f46a3e7040\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " Oct 31 19:54:12.467534 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467389 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-envoy\") pod \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " Oct 31 19:54:12.467534 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467426 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-certs\") pod \"b8691586-d76a-495b-9515-99f46a3e7040\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " Oct 31 19:54:12.467534 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467453 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-data" pod "b8691586-d76a-495b-9515-99f46a3e7040" (UID: "b8691586-d76a-495b-9515-99f46a3e7040"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.467534 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467460 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-credential-socket\") pod \"b8691586-d76a-495b-9515-99f46a3e7040\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " Oct 31 19:54:12.467534 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467489 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-socket\") pod \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " Oct 31 19:54:12.467790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467545 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-data\") pod \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " Oct 31 19:54:12.467790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467569 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8691586-d76a-495b-9515-99f46a3e7040-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8691586-d76a-495b-9515-99f46a3e7040-istiod-ca-cert\") pod \"b8691586-d76a-495b-9515-99f46a3e7040\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " Oct 31 19:54:12.467790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467595 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8691586-d76a-495b-9515-99f46a3e7040-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8691586-d76a-495b-9515-99f46a3e7040-istio-podinfo\") pod \"b8691586-d76a-495b-9515-99f46a3e7040\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " Oct 31 19:54:12.467790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467623 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-certs\") pod \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " Oct 31 19:54:12.467790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467684 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-socket" pod "b8691586-d76a-495b-9515-99f46a3e7040" (UID: "b8691586-d76a-495b-9515-99f46a3e7040"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.467790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467695 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-certs" pod "b8691586-d76a-495b-9515-99f46a3e7040" (UID: "b8691586-d76a-495b-9515-99f46a3e7040"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.467790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467711 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-credential-socket\") pod \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " Oct 31 19:54:12.468166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467795 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-credential-socket" pod "b8691586-d76a-495b-9515-99f46a3e7040" (UID: "b8691586-d76a-495b-9515-99f46a3e7040"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.468166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467831 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-token\" (UniqueName: \"kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-token\") pod \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " Oct 31 19:54:12.468166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467876 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-data\") pod \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " Oct 31 19:54:12.468166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467941 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-podinfo\") pod \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " Oct 31 19:54:12.468166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467950 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-credential-socket" pod "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" (UID: "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.468166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.467984 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istiod-ca-cert\") pod \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " Oct 31 19:54:12.468166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468023 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-envoy\") pod \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " Oct 31 19:54:12.468166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468079 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istiod-ca-cert\") pod \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " Oct 31 19:54:12.468166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468112 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-certs\") pod \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " Oct 31 19:54:12.468166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468103 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-certs" pod "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" (UID: "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.468166 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468156 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-kube-api-access-c8qj6\" (UniqueName: \"kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-kube-api-access-c8qj6\") pod \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " Oct 31 19:54:12.468744 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468190 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-kube-api-access-tj9g2\" (UniqueName: \"kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-kube-api-access-tj9g2\") pod \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " Oct 31 19:54:12.468744 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468221 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-socket\") pod \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " Oct 31 19:54:12.468744 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468247 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-token\" (UniqueName: \"kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-token\") pod \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\" (UID: \"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e\") " Oct 31 19:54:12.468744 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468273 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-kube-api-access-ph7ln\" (UniqueName: \"kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-kube-api-access-ph7ln\") pod \"b8691586-d76a-495b-9515-99f46a3e7040\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " Oct 31 19:54:12.468744 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468320 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-istio-token\" (UniqueName: \"kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-istio-token\") pod \"b8691586-d76a-495b-9515-99f46a3e7040\" (UID: \"b8691586-d76a-495b-9515-99f46a3e7040\") " Oct 31 19:54:12.468744 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468359 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-credential-socket\") pod \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\" (UID: \"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68\") " Oct 31 19:54:12.469194 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.469169 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.469324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.469199 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.469324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.469214 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.469324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.469229 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.469324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.469242 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.469324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.469256 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.469594 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468155 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-socket" pod "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" (UID: "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.469594 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468341 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-data" pod "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" (UID: "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.469594 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468559 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istiod-ca-cert" pod "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" (UID: "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:12.469594 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.468637 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istiod-ca-cert" pod "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" (UID: "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:12.469594 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.469131 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-data" pod "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" (UID: "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.469594 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.469316 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-certs" pod "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" (UID: "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.469594 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.469432 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-credential-socket" pod "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" (UID: "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.470053 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.469603 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-socket" pod "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" (UID: "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.470801 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.470770 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8691586-d76a-495b-9515-99f46a3e7040-istiod-ca-cert" pod "b8691586-d76a-495b-9515-99f46a3e7040" (UID: "b8691586-d76a-495b-9515-99f46a3e7040"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:12.471550 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.471509 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-podinfo" pod "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" (UID: "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:12.471550 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.471509 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-podinfo" pod "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" (UID: "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:12.472041 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.472020 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-kube-api-access-tj9g2" pod "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" (UID: "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e"). InnerVolumeSpecName "kube-api-access-tj9g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:12.472632 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.472604 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-token" pod "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" (UID: "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:12.472821 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.472795 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-envoy" pod "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" (UID: "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.472821 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.472806 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-envoy" pod "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" (UID: "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.473043 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.472960 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-envoy" pod "b8691586-d76a-495b-9515-99f46a3e7040" (UID: "b8691586-d76a-495b-9515-99f46a3e7040"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.473342 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.473309 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-istio-token" pod "b8691586-d76a-495b-9515-99f46a3e7040" (UID: "b8691586-d76a-495b-9515-99f46a3e7040"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:12.473430 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.473384 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8691586-d76a-495b-9515-99f46a3e7040-istio-podinfo" pod "b8691586-d76a-495b-9515-99f46a3e7040" (UID: "b8691586-d76a-495b-9515-99f46a3e7040"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:12.473711 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.473693 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-token" pod "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" (UID: "7528b131-3ca2-4da8-a966-bc6a1e8f0d2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:12.473823 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.473803 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-kube-api-access-ph7ln" pod "b8691586-d76a-495b-9515-99f46a3e7040" (UID: "b8691586-d76a-495b-9515-99f46a3e7040"). InnerVolumeSpecName "kube-api-access-ph7ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:12.474657 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.474637 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-kube-api-access-c8qj6" pod "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" (UID: "2d9aad8c-a2ad-4989-a088-4cda2f8a2a68"). InnerVolumeSpecName "kube-api-access-c8qj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:12.571058 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571022 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8691586-d76a-495b-9515-99f46a3e7040-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571058 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571057 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571074 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571088 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571103 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8691586-d76a-495b-9515-99f46a3e7040-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571116 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8691586-d76a-495b-9515-99f46a3e7040-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571129 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571143 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571157 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571171 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571185 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571198 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571212 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571226 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8qj6\" (UniqueName: \"kubernetes.io/projected/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-kube-api-access-c8qj6\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571243 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj9g2\" (UniqueName: \"kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-kube-api-access-tj9g2\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571261 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571278 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571293 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7ln\" (UniqueName: \"kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-kube-api-access-ph7ln\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571307 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8691586-d76a-495b-9515-99f46a3e7040-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571320 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.571327 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.571328 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:12.631939 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.631738 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj"] Oct 31 19:54:12.638125 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:12.638095 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-92gxj"] Oct 31 19:54:13.375909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.374795 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" Oct 31 19:54:13.375909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.375241 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh" event={"ID":"b8691586-d76a-495b-9515-99f46a3e7040","Type":"ContainerDied","Data":"12cfdea0b627caff6f2b8897242358fa2c12c439794377bdc696ac860faf21c1"} Oct 31 19:54:13.375909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.375298 6107 scope.go:122] "RemoveContainer" containerID="1e55fd5efa74f3359af678cb8db55a476c6162f41590098bf94d6e0948db9134" Oct 31 19:54:13.382374 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.382337 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" event={"ID":"2d9aad8c-a2ad-4989-a088-4cda2f8a2a68","Type":"ContainerDied","Data":"75f985a13dbc2318a59422de867e72919b697fbef4d193fcd28555b88b9b8254"} Oct 31 19:54:13.382507 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.382466 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch" Oct 31 19:54:13.387748 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.387723 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" event={"ID":"7528b131-3ca2-4da8-a966-bc6a1e8f0d2e","Type":"ContainerDied","Data":"f6d23ab4b96b0491eaffa569962f52585697336b1b1d2c347a9b2cfd1a3e047f"} Oct 31 19:54:13.387921 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.387747 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh" Oct 31 19:54:13.390846 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.390826 6107 scope.go:122] "RemoveContainer" containerID="501b362e65d3d431d0d1dabc4d3d819fae7fb9b1bf8cc3892d638f3ef1a3178f" Oct 31 19:54:13.401474 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.401450 6107 scope.go:122] "RemoveContainer" containerID="0dd0ee4ea2f520ec9e9ba8c60af8032a473ad06920744115659595ce36d58d30" Oct 31 19:54:13.600704 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.600676 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch"] Oct 31 19:54:13.612651 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.610487 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-kcpch"] Oct 31 19:54:13.726023 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.725987 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh"] Oct 31 19:54:13.729449 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.729398 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-nj2gh"] Oct 31 19:54:13.831993 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.831956 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh"] Oct 31 19:54:13.838162 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:13.838093 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-8zbwh"] Oct 31 19:54:14.495652 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:14.494798 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" path="/var/lib/kubelet/pods/2d9aad8c-a2ad-4989-a088-4cda2f8a2a68/volumes" Oct 31 19:54:14.495652 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:14.495616 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" path="/var/lib/kubelet/pods/7528b131-3ca2-4da8-a966-bc6a1e8f0d2e/volumes" Oct 31 19:54:14.496407 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:14.496284 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8691586-d76a-495b-9515-99f46a3e7040" path="/var/lib/kubelet/pods/b8691586-d76a-495b-9515-99f46a3e7040/volumes" Oct 31 19:54:14.497143 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:14.497124 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" path="/var/lib/kubelet/pods/f9a7ba14-180b-4ee3-9917-e13736fd6ad2/volumes" Oct 31 19:54:15.770871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:15.770801 6107 patch_prober.go:29] interesting pod/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:54:15.770871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:15.770868 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:54:17.108308 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.108277 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.248659 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-credential-socket\") pod \"47e05a2c-520b-46a6-80cb-60672de0f2c9\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.248732 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-envoy\") pod \"47e05a2c-520b-46a6-80cb-60672de0f2c9\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.248764 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-kube-api-access-ch25v\" (UniqueName: \"kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-kube-api-access-ch25v\") pod \"47e05a2c-520b-46a6-80cb-60672de0f2c9\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.248938 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-podinfo\") pod \"47e05a2c-520b-46a6-80cb-60672de0f2c9\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.249039 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e05a2c-520b-46a6-80cb-60672de0f2c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e05a2c-520b-46a6-80cb-60672de0f2c9-istiod-ca-cert\") pod \"47e05a2c-520b-46a6-80cb-60672de0f2c9\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.249116 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-socket\") pod \"47e05a2c-520b-46a6-80cb-60672de0f2c9\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.249203 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-certs\") pod \"47e05a2c-520b-46a6-80cb-60672de0f2c9\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.249239 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-token\" (UniqueName: \"kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-token\") pod \"47e05a2c-520b-46a6-80cb-60672de0f2c9\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.249315 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-data\") pod \"47e05a2c-520b-46a6-80cb-60672de0f2c9\" (UID: \"47e05a2c-520b-46a6-80cb-60672de0f2c9\") " Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.250308 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-data" pod "47e05a2c-520b-46a6-80cb-60672de0f2c9" (UID: "47e05a2c-520b-46a6-80cb-60672de0f2c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.250526 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-credential-socket" pod "47e05a2c-520b-46a6-80cb-60672de0f2c9" (UID: "47e05a2c-520b-46a6-80cb-60672de0f2c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.251090 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-certs" pod "47e05a2c-520b-46a6-80cb-60672de0f2c9" (UID: "47e05a2c-520b-46a6-80cb-60672de0f2c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:17.253902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.251259 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-socket" pod "47e05a2c-520b-46a6-80cb-60672de0f2c9" (UID: "47e05a2c-520b-46a6-80cb-60672de0f2c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:17.255509 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.255471 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e05a2c-520b-46a6-80cb-60672de0f2c9-istiod-ca-cert" pod "47e05a2c-520b-46a6-80cb-60672de0f2c9" (UID: "47e05a2c-520b-46a6-80cb-60672de0f2c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:17.290318 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.290257 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-kube-api-access-ch25v" pod "47e05a2c-520b-46a6-80cb-60672de0f2c9" (UID: "47e05a2c-520b-46a6-80cb-60672de0f2c9"). InnerVolumeSpecName "kube-api-access-ch25v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:17.294027 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.293992 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-envoy" pod "47e05a2c-520b-46a6-80cb-60672de0f2c9" (UID: "47e05a2c-520b-46a6-80cb-60672de0f2c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:17.294178 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.294147 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-token" pod "47e05a2c-520b-46a6-80cb-60672de0f2c9" (UID: "47e05a2c-520b-46a6-80cb-60672de0f2c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:17.354575 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.354429 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:17.354802 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.354783 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:17.354923 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.354908 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch25v\" (UniqueName: \"kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-kube-api-access-ch25v\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:17.355029 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.355018 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e05a2c-520b-46a6-80cb-60672de0f2c9-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:17.355119 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.355108 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:17.355200 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.355189 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:17.355275 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.355264 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:17.355349 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.355339 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:17.375905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.374434 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-podinfo" pod "47e05a2c-520b-46a6-80cb-60672de0f2c9" (UID: "47e05a2c-520b-46a6-80cb-60672de0f2c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:17.436904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.436473 6107 generic.go:356] "Generic (PLEG): container finished" podID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerID="dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909" exitCode=0 Oct 31 19:54:17.436904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.436542 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" event={"ID":"47e05a2c-520b-46a6-80cb-60672de0f2c9","Type":"ContainerDied","Data":"dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909"} Oct 31 19:54:17.436904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.436583 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" Oct 31 19:54:17.436904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.436601 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx" event={"ID":"47e05a2c-520b-46a6-80cb-60672de0f2c9","Type":"ContainerDied","Data":"9cecb29f2944c0e644880f734fee857c24d235415c502c48e86dad32ef269530"} Oct 31 19:54:17.436904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.436631 6107 scope.go:122] "RemoveContainer" containerID="dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909" Oct 31 19:54:17.455942 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.455871 6107 scope.go:122] "RemoveContainer" containerID="a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3" Oct 31 19:54:17.457262 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.457241 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e05a2c-520b-46a6-80cb-60672de0f2c9-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:17.468024 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.467991 6107 scope.go:122] "RemoveContainer" containerID="dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909" Oct 31 19:54:17.468334 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:54:17.468306 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909\": container with ID starting with dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909 not found: ID does not exist" containerID="dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909" Oct 31 19:54:17.468440 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.468335 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909"} err="failed to get container status \"dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909\": rpc error: code = NotFound desc = could not find container \"dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909\": container with ID starting with dd21c231dddd204b11f94438765606ee54df561051418091e935ca3bb4cc3909 not found: ID does not exist" Oct 31 19:54:17.468440 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.468353 6107 scope.go:122] "RemoveContainer" containerID="a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3" Oct 31 19:54:17.468630 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:54:17.468608 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3\": container with ID starting with a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3 not found: ID does not exist" containerID="a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3" Oct 31 19:54:17.468712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.468637 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3"} err="failed to get container status \"a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3\": rpc error: code = NotFound desc = could not find container \"a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3\": container with ID starting with a65992823d6d94437128406c184f58b7362b5c39a8db92860778242ff4fff8e3 not found: ID does not exist" Oct 31 19:54:17.591499 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.591464 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx"] Oct 31 19:54:17.595074 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:17.595048 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-abb7688a-gw-openshift-default-558d66676f-zksnx"] Oct 31 19:54:18.488647 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:18.488612 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" path="/var/lib/kubelet/pods/47e05a2c-520b-46a6-80cb-60672de0f2c9/volumes" Oct 31 19:54:20.406391 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:54:20.406349 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:54:22.858051 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:22.858014 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7"] Oct 31 19:54:22.858547 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:22.858286 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" containerID="cri-o://9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a" gracePeriod=30 Oct 31 19:54:22.880141 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:22.880096 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb"] Oct 31 19:54:22.880431 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:22.880397 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" containerID="cri-o://e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953" gracePeriod=30 Oct 31 19:54:27.970814 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:54:27.970780 6107 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f67579_7b35_47c7_ae72_127726600e78.slice/crio-9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a.scope\": RecentStats: unable to find data in memory cache]" Oct 31 19:54:28.163332 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.163299 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:54:28.204994 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.204966 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-data\") pod \"70f67579-7b35-47c7-ae72-127726600e78\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " Oct 31 19:54:28.205139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.205039 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-credential-socket\") pod \"70f67579-7b35-47c7-ae72-127726600e78\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " Oct 31 19:54:28.205139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.205069 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-socket\") pod \"70f67579-7b35-47c7-ae72-127726600e78\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " Oct 31 19:54:28.205139 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.205110 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-envoy\") pod \"70f67579-7b35-47c7-ae72-127726600e78\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " Oct 31 19:54:28.205310 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.205197 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-istio-token\" (UniqueName: \"kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-istio-token\") pod \"70f67579-7b35-47c7-ae72-127726600e78\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " Oct 31 19:54:28.205310 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.205219 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-data" pod "70f67579-7b35-47c7-ae72-127726600e78" (UID: "70f67579-7b35-47c7-ae72-127726600e78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:28.205310 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.205238 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-kube-api-access-d9cnn\" (UniqueName: \"kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-kube-api-access-d9cnn\") pod \"70f67579-7b35-47c7-ae72-127726600e78\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " Oct 31 19:54:28.205483 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.205321 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70f67579-7b35-47c7-ae72-127726600e78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f67579-7b35-47c7-ae72-127726600e78-istio-podinfo\") pod \"70f67579-7b35-47c7-ae72-127726600e78\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " Oct 31 19:54:28.205483 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.205337 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-credential-socket" pod "70f67579-7b35-47c7-ae72-127726600e78" (UID: "70f67579-7b35-47c7-ae72-127726600e78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:28.205483 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.205376 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-socket" pod "70f67579-7b35-47c7-ae72-127726600e78" (UID: "70f67579-7b35-47c7-ae72-127726600e78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:28.205483 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.205474 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f67579-7b35-47c7-ae72-127726600e78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f67579-7b35-47c7-ae72-127726600e78-istiod-ca-cert\") pod \"70f67579-7b35-47c7-ae72-127726600e78\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " Oct 31 19:54:28.205708 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.205508 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-certs\") pod \"70f67579-7b35-47c7-ae72-127726600e78\" (UID: \"70f67579-7b35-47c7-ae72-127726600e78\") " Oct 31 19:54:28.206181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.206158 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.206181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.206175 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f67579-7b35-47c7-ae72-127726600e78-istiod-ca-cert" pod "70f67579-7b35-47c7-ae72-127726600e78" (UID: "70f67579-7b35-47c7-ae72-127726600e78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:28.206324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.206185 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.206324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.206201 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.206440 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.206402 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-certs" pod "70f67579-7b35-47c7-ae72-127726600e78" (UID: "70f67579-7b35-47c7-ae72-127726600e78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:28.207673 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.207647 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-kube-api-access-d9cnn" pod "70f67579-7b35-47c7-ae72-127726600e78" (UID: "70f67579-7b35-47c7-ae72-127726600e78"). InnerVolumeSpecName "kube-api-access-d9cnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:28.207772 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.207702 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-istio-token" pod "70f67579-7b35-47c7-ae72-127726600e78" (UID: "70f67579-7b35-47c7-ae72-127726600e78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:28.207972 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.207940 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-envoy" pod "70f67579-7b35-47c7-ae72-127726600e78" (UID: "70f67579-7b35-47c7-ae72-127726600e78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:28.208235 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.208208 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70f67579-7b35-47c7-ae72-127726600e78-istio-podinfo" pod "70f67579-7b35-47c7-ae72-127726600e78" (UID: "70f67579-7b35-47c7-ae72-127726600e78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:28.217235 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.217211 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:54:28.307267 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307234 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-socket\") pod \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " Oct 31 19:54:28.307446 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307331 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-podinfo\") pod \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " Oct 31 19:54:28.307446 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307359 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-credential-socket\") pod \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " Oct 31 19:54:28.307446 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307409 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-token\" (UniqueName: \"kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-token\") pod \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " Oct 31 19:54:28.307620 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307446 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-certs\") pod \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " Oct 31 19:54:28.307620 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307536 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-socket" pod "1dbd00aa-2a24-4e6c-bba8-2b66287efede" (UID: "1dbd00aa-2a24-4e6c-bba8-2b66287efede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:28.307620 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307582 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istiod-ca-cert\") pod \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " Oct 31 19:54:28.307784 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307636 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-kube-api-access-rqpq2\" (UniqueName: \"kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-kube-api-access-rqpq2\") pod \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " Oct 31 19:54:28.307784 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307644 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-credential-socket" pod "1dbd00aa-2a24-4e6c-bba8-2b66287efede" (UID: "1dbd00aa-2a24-4e6c-bba8-2b66287efede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:28.307784 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307678 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-data\") pod \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " Oct 31 19:54:28.307784 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307768 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-envoy\") pod \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\" (UID: \"1dbd00aa-2a24-4e6c-bba8-2b66287efede\") " Oct 31 19:54:28.308018 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307797 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-certs" pod "1dbd00aa-2a24-4e6c-bba8-2b66287efede" (UID: "1dbd00aa-2a24-4e6c-bba8-2b66287efede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:28.308018 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.307925 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istiod-ca-cert" pod "1dbd00aa-2a24-4e6c-bba8-2b66287efede" (UID: "1dbd00aa-2a24-4e6c-bba8-2b66287efede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:28.308126 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.308066 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-data" pod "1dbd00aa-2a24-4e6c-bba8-2b66287efede" (UID: "1dbd00aa-2a24-4e6c-bba8-2b66287efede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:28.308396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.308374 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.308396 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.308396 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.308560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.308411 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9cnn\" (UniqueName: \"kubernetes.io/projected/70f67579-7b35-47c7-ae72-127726600e78-kube-api-access-d9cnn\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.308560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.308426 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f67579-7b35-47c7-ae72-127726600e78-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.308560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.308440 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.308560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.308453 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f67579-7b35-47c7-ae72-127726600e78-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.308560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.308466 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.308560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.308474 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.308560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.308482 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.308560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.308492 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f67579-7b35-47c7-ae72-127726600e78-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.308560 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.308501 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.309940 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.309875 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-kube-api-access-rqpq2" pod "1dbd00aa-2a24-4e6c-bba8-2b66287efede" (UID: "1dbd00aa-2a24-4e6c-bba8-2b66287efede"). InnerVolumeSpecName "kube-api-access-rqpq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:28.310091 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.310073 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-podinfo" pod "1dbd00aa-2a24-4e6c-bba8-2b66287efede" (UID: "1dbd00aa-2a24-4e6c-bba8-2b66287efede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:28.310159 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.310137 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-token" pod "1dbd00aa-2a24-4e6c-bba8-2b66287efede" (UID: "1dbd00aa-2a24-4e6c-bba8-2b66287efede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:28.310266 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.310252 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-envoy" pod "1dbd00aa-2a24-4e6c-bba8-2b66287efede" (UID: "1dbd00aa-2a24-4e6c-bba8-2b66287efede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:28.410141 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.410059 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.410141 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.410087 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.410141 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.410097 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqpq2\" (UniqueName: \"kubernetes.io/projected/1dbd00aa-2a24-4e6c-bba8-2b66287efede-kube-api-access-rqpq2\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.410141 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.410107 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbd00aa-2a24-4e6c-bba8-2b66287efede-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:28.544090 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.543939 6107 generic.go:356] "Generic (PLEG): container finished" podID="70f67579-7b35-47c7-ae72-127726600e78" containerID="9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a" exitCode=0 Oct 31 19:54:28.544090 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.544003 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" Oct 31 19:54:28.544090 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.544070 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" event={"ID":"70f67579-7b35-47c7-ae72-127726600e78","Type":"ContainerDied","Data":"9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a"} Oct 31 19:54:28.544488 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.544118 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7" event={"ID":"70f67579-7b35-47c7-ae72-127726600e78","Type":"ContainerDied","Data":"85bcf6731282dfc33d81fb4ed64acb16a425fc20d7419f01d23ebfa28b6ca202"} Oct 31 19:54:28.544488 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.544148 6107 scope.go:122] "RemoveContainer" containerID="9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a" Oct 31 19:54:28.546576 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.546536 6107 generic.go:356] "Generic (PLEG): container finished" podID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerID="e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953" exitCode=0 Oct 31 19:54:28.546721 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.546702 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" event={"ID":"1dbd00aa-2a24-4e6c-bba8-2b66287efede","Type":"ContainerDied","Data":"e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953"} Oct 31 19:54:28.546788 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.546746 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" Oct 31 19:54:28.546857 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.546746 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb" event={"ID":"1dbd00aa-2a24-4e6c-bba8-2b66287efede","Type":"ContainerDied","Data":"acf38485eaf14844e785945a024d44468d856ff032b14274fd3899c974c16260"} Oct 31 19:54:28.556105 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.556081 6107 scope.go:122] "RemoveContainer" containerID="9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a" Oct 31 19:54:28.556386 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:54:28.556367 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a\": container with ID starting with 9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a not found: ID does not exist" containerID="9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a" Oct 31 19:54:28.556457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.556395 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a"} err="failed to get container status \"9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a\": rpc error: code = NotFound desc = could not find container \"9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a\": container with ID starting with 9bf973dc874074eeb41e7e1417282236470c96239ec235367b60495cf9f4636a not found: ID does not exist" Oct 31 19:54:28.556457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.556417 6107 scope.go:122] "RemoveContainer" containerID="e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953" Oct 31 19:54:28.568132 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.568110 6107 scope.go:122] "RemoveContainer" containerID="e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953" Oct 31 19:54:28.568414 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:54:28.568391 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953\": container with ID starting with e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953 not found: ID does not exist" containerID="e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953" Oct 31 19:54:28.568485 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.568421 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953"} err="failed to get container status \"e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953\": rpc error: code = NotFound desc = could not find container \"e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953\": container with ID starting with e1411bb9bd0a52f1f02770317d41d120a5eaeefdc39bdb308ca3985768f19953 not found: ID does not exist" Oct 31 19:54:28.708792 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.708691 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb"] Oct 31 19:54:28.712478 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.712452 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-j9pjb"] Oct 31 19:54:28.812561 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.812521 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7"] Oct 31 19:54:28.819256 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:28.819231 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-qh6s7"] Oct 31 19:54:30.490681 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:30.490631 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" path="/var/lib/kubelet/pods/1dbd00aa-2a24-4e6c-bba8-2b66287efede/volumes" Oct 31 19:54:30.491695 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:54:30.491674 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f67579-7b35-47c7-ae72-127726600e78" path="/var/lib/kubelet/pods/70f67579-7b35-47c7-ae72-127726600e78/volumes" Oct 31 19:54:50.409929 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:54:50.409835 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:55:20.405554 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:55:20.405513 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:55:34.282411 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.282375 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6"] Oct 31 19:55:34.283046 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283030 6107 cpu_manager.go:414] "containerMap: removing container" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" Oct 31 19:55:34.283101 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283047 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" Oct 31 19:55:34.283101 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283075 6107 cpu_manager.go:414] "containerMap: removing container" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" Oct 31 19:55:34.283101 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283080 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" Oct 31 19:55:34.283197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283103 6107 cpu_manager.go:414] "containerMap: removing container" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerName="istio-proxy" Oct 31 19:55:34.283197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283108 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerName="istio-proxy" Oct 31 19:55:34.283197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283116 6107 cpu_manager.go:414] "containerMap: removing container" podUID="b8691586-d76a-495b-9515-99f46a3e7040" containerName="istio-proxy" Oct 31 19:55:34.283197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283121 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8691586-d76a-495b-9515-99f46a3e7040" containerName="istio-proxy" Oct 31 19:55:34.283197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283130 6107 cpu_manager.go:414] "containerMap: removing container" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" Oct 31 19:55:34.283197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283134 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" Oct 31 19:55:34.283197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283144 6107 cpu_manager.go:414] "containerMap: removing container" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" Oct 31 19:55:34.283197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283148 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" Oct 31 19:55:34.283197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283158 6107 cpu_manager.go:414] "containerMap: removing container" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" Oct 31 19:55:34.283197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283163 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" Oct 31 19:55:34.283483 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283272 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbd00aa-2a24-4e6c-bba8-2b66287efede" containerName="istio-proxy" Oct 31 19:55:34.283483 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283283 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerName="istio-proxy" Oct 31 19:55:34.283483 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283289 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a7ba14-180b-4ee3-9917-e13736fd6ad2" containerName="istio-proxy" Oct 31 19:55:34.283483 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283297 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" Oct 31 19:55:34.283483 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283305 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" Oct 31 19:55:34.283483 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283312 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="7528b131-3ca2-4da8-a966-bc6a1e8f0d2e" containerName="istio-proxy" Oct 31 19:55:34.283483 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283319 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8691586-d76a-495b-9515-99f46a3e7040" containerName="istio-proxy" Oct 31 19:55:34.283483 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.283324 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f67579-7b35-47c7-ae72-127726600e78" containerName="istio-proxy" Oct 31 19:55:34.286827 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.286809 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" Oct 31 19:55:34.339486 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.339453 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6"] Oct 31 19:55:34.387016 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.386982 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5370f5\"" Oct 31 19:55:34.414927 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.411425 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5370f5-dockercfg-wljl6\"" Oct 31 19:55:34.442544 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.442504 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smnx6\" (UniqueName: \"kubernetes.io/projected/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-kube-api-access-smnx6\") pod \"payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6\" (UID: \"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" Oct 31 19:55:34.443061 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.443033 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6\" (UID: \"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" Oct 31 19:55:34.450067 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.450040 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6"] Oct 31 19:55:34.451260 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.451232 6107 cpu_manager.go:414] "containerMap: removing container" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" Oct 31 19:55:34.451433 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.451418 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e05a2c-520b-46a6-80cb-60672de0f2c9" containerName="istio-proxy" Oct 31 19:55:34.451588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.451577 6107 cpu_manager.go:414] "containerMap: removing container" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerName="istio-proxy" Oct 31 19:55:34.451725 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.451710 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerName="istio-proxy" Oct 31 19:55:34.452948 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.452006 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9aad8c-a2ad-4989-a088-4cda2f8a2a68" containerName="istio-proxy" Oct 31 19:55:34.469790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.468478 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" Oct 31 19:55:34.550876 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.550385 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smnx6\" (UniqueName: \"kubernetes.io/projected/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-kube-api-access-smnx6\") pod \"payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6\" (UID: \"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" Oct 31 19:55:34.550876 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.550541 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6\" (UID: \"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" Oct 31 19:55:34.550876 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.550600 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b851ea3a-e54b-4119-b2d7-5452084005f2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6\" (UID: \"b851ea3a-e54b-4119-b2d7-5452084005f2\") " pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" Oct 31 19:55:34.550876 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.550630 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrpv\" (UniqueName: \"kubernetes.io/projected/b851ea3a-e54b-4119-b2d7-5452084005f2-kube-api-access-tjrpv\") pod \"payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6\" (UID: \"b851ea3a-e54b-4119-b2d7-5452084005f2\") " pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" Oct 31 19:55:34.551823 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.551789 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6\" (UID: \"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" Oct 31 19:55:34.574054 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.574022 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6"] Oct 31 19:55:34.605389 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.605357 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smnx6\" (UniqueName: \"kubernetes.io/projected/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-kube-api-access-smnx6\") pod \"payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6\" (UID: \"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" Oct 31 19:55:34.651890 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.651843 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b851ea3a-e54b-4119-b2d7-5452084005f2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6\" (UID: \"b851ea3a-e54b-4119-b2d7-5452084005f2\") " pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" Oct 31 19:55:34.651890 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.651895 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrpv\" (UniqueName: \"kubernetes.io/projected/b851ea3a-e54b-4119-b2d7-5452084005f2-kube-api-access-tjrpv\") pod \"payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6\" (UID: \"b851ea3a-e54b-4119-b2d7-5452084005f2\") " pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" Oct 31 19:55:34.652754 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.652731 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b851ea3a-e54b-4119-b2d7-5452084005f2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6\" (UID: \"b851ea3a-e54b-4119-b2d7-5452084005f2\") " pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" Oct 31 19:55:34.705374 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.705344 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrpv\" (UniqueName: \"kubernetes.io/projected/b851ea3a-e54b-4119-b2d7-5452084005f2-kube-api-access-tjrpv\") pod \"payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6\" (UID: \"b851ea3a-e54b-4119-b2d7-5452084005f2\") " pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" Oct 31 19:55:34.882740 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.882660 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" Oct 31 19:55:34.901874 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:34.901846 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" Oct 31 19:55:35.265175 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:35.265144 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6"] Oct 31 19:55:35.265768 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:55:35.265744 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb851ea3a_e54b_4119_b2d7_5452084005f2.slice/crio-a526fb87183e07de5dd31b75f41fc550080e8ec8d5a155490bfc8b43b60c0b3c WatchSource:0}: Error finding container a526fb87183e07de5dd31b75f41fc550080e8ec8d5a155490bfc8b43b60c0b3c: Status 404 returned error can't find the container with id a526fb87183e07de5dd31b75f41fc550080e8ec8d5a155490bfc8b43b60c0b3c Oct 31 19:55:35.311197 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:35.311172 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6"] Oct 31 19:55:35.313767 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:55:35.313738 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6428a1a_6ae9_4fb6_bb42_16e7dd39bf62.slice/crio-c75ceab1743a1f29ccb341024e2abc54be142def26b98da8efb9178d74ef2f4f WatchSource:0}: Error finding container c75ceab1743a1f29ccb341024e2abc54be142def26b98da8efb9178d74ef2f4f: Status 404 returned error can't find the container with id c75ceab1743a1f29ccb341024e2abc54be142def26b98da8efb9178d74ef2f4f Oct 31 19:55:36.162130 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:36.162086 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" event={"ID":"b851ea3a-e54b-4119-b2d7-5452084005f2","Type":"ContainerStarted","Data":"01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444"} Oct 31 19:55:36.162328 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:36.162191 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" Oct 31 19:55:36.162328 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:36.162219 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" event={"ID":"b851ea3a-e54b-4119-b2d7-5452084005f2","Type":"ContainerStarted","Data":"a526fb87183e07de5dd31b75f41fc550080e8ec8d5a155490bfc8b43b60c0b3c"} Oct 31 19:55:36.163952 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:36.163924 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" event={"ID":"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62","Type":"ContainerStarted","Data":"88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d"} Oct 31 19:55:36.164075 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:36.163967 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" event={"ID":"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62","Type":"ContainerStarted","Data":"c75ceab1743a1f29ccb341024e2abc54be142def26b98da8efb9178d74ef2f4f"} Oct 31 19:55:36.164075 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:36.163987 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" Oct 31 19:55:36.270742 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:36.270686 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" podStartSLOduration=2.270671575 podStartE2EDuration="2.270671575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:36.269573777 +0000 UTC m=+2307.902967685" watchObservedRunningTime="2026-10-31 19:55:36.270671575 +0000 UTC m=+2307.904065474" Oct 31 19:55:36.380369 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:36.379341 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" podStartSLOduration=2.379321466 podStartE2EDuration="2.379321466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:36.375435293 +0000 UTC m=+2308.008829202" watchObservedRunningTime="2026-10-31 19:55:36.379321466 +0000 UTC m=+2308.012715375" Oct 31 19:55:47.174731 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:47.174701 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" Oct 31 19:55:47.175293 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:55:47.174768 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" Oct 31 19:55:50.406796 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:55:50.406747 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:56:08.584330 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.584298 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6"] Oct 31 19:56:08.589033 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.589007 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" Oct 31 19:56:08.614316 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.614282 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2gsb\" (UniqueName: \"kubernetes.io/projected/c17338a4-ac0b-4b76-a622-28fe74269fa9-kube-api-access-s2gsb\") pod \"payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6\" (UID: \"c17338a4-ac0b-4b76-a622-28fe74269fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" Oct 31 19:56:08.614499 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.614343 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17338a4-ac0b-4b76-a622-28fe74269fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6\" (UID: \"c17338a4-ac0b-4b76-a622-28fe74269fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" Oct 31 19:56:08.724278 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.724245 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-9a8346\"" Oct 31 19:56:08.724824 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.724799 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-9a8346-dockercfg-gshpc\"" Oct 31 19:56:08.734090 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.734045 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6"] Oct 31 19:56:08.739296 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.739269 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5"] Oct 31 19:56:08.772155 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.757830 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2gsb\" (UniqueName: \"kubernetes.io/projected/c17338a4-ac0b-4b76-a622-28fe74269fa9-kube-api-access-s2gsb\") pod \"payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6\" (UID: \"c17338a4-ac0b-4b76-a622-28fe74269fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" Oct 31 19:56:08.772155 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.757934 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17338a4-ac0b-4b76-a622-28fe74269fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6\" (UID: \"c17338a4-ac0b-4b76-a622-28fe74269fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" Oct 31 19:56:08.772155 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.759060 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17338a4-ac0b-4b76-a622-28fe74269fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6\" (UID: \"c17338a4-ac0b-4b76-a622-28fe74269fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" Oct 31 19:56:08.778500 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.775170 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" Oct 31 19:56:08.880181 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.879908 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5"] Oct 31 19:56:08.894739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.894706 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2gsb\" (UniqueName: \"kubernetes.io/projected/c17338a4-ac0b-4b76-a622-28fe74269fa9-kube-api-access-s2gsb\") pod \"payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6\" (UID: \"c17338a4-ac0b-4b76-a622-28fe74269fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" Oct 31 19:56:08.904206 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.904170 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" Oct 31 19:56:08.977626 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.977595 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d03485-903b-480b-a923-08a9fd632e7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5\" (UID: \"f7d03485-903b-480b-a923-08a9fd632e7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" Oct 31 19:56:08.977811 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:08.977769 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hpk8\" (UniqueName: \"kubernetes.io/projected/f7d03485-903b-480b-a923-08a9fd632e7b-kube-api-access-7hpk8\") pod \"payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5\" (UID: \"f7d03485-903b-480b-a923-08a9fd632e7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" Oct 31 19:56:09.080091 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:09.080034 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d03485-903b-480b-a923-08a9fd632e7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5\" (UID: \"f7d03485-903b-480b-a923-08a9fd632e7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" Oct 31 19:56:09.080345 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:09.080215 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hpk8\" (UniqueName: \"kubernetes.io/projected/f7d03485-903b-480b-a923-08a9fd632e7b-kube-api-access-7hpk8\") pod \"payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5\" (UID: \"f7d03485-903b-480b-a923-08a9fd632e7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" Oct 31 19:56:09.081397 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:09.081366 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d03485-903b-480b-a923-08a9fd632e7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5\" (UID: \"f7d03485-903b-480b-a923-08a9fd632e7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" Oct 31 19:56:09.134046 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:09.133973 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hpk8\" (UniqueName: \"kubernetes.io/projected/f7d03485-903b-480b-a923-08a9fd632e7b-kube-api-access-7hpk8\") pod \"payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5\" (UID: \"f7d03485-903b-480b-a923-08a9fd632e7b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" Oct 31 19:56:09.205859 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:09.205826 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" Oct 31 19:56:09.308370 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:09.308339 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6"] Oct 31 19:56:09.308952 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:56:09.308921 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17338a4_ac0b_4b76_a622_28fe74269fa9.slice/crio-73f387952d794ef4161f9f0ac9d1799acf7c366019c25bd36d95193272b835eb WatchSource:0}: Error finding container 73f387952d794ef4161f9f0ac9d1799acf7c366019c25bd36d95193272b835eb: Status 404 returned error can't find the container with id 73f387952d794ef4161f9f0ac9d1799acf7c366019c25bd36d95193272b835eb Oct 31 19:56:09.477710 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:09.477672 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" event={"ID":"c17338a4-ac0b-4b76-a622-28fe74269fa9","Type":"ContainerStarted","Data":"54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878"} Oct 31 19:56:09.477896 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:09.477725 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" event={"ID":"c17338a4-ac0b-4b76-a622-28fe74269fa9","Type":"ContainerStarted","Data":"73f387952d794ef4161f9f0ac9d1799acf7c366019c25bd36d95193272b835eb"} Oct 31 19:56:09.477896 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:09.477748 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" Oct 31 19:56:09.586076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:09.586042 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5"] Oct 31 19:56:09.586520 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:56:09.586326 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d03485_903b_480b_a923_08a9fd632e7b.slice/crio-0ad8adbaf6b4ff768891c74f1a1809de290c4343913178719e7db009540ff271 WatchSource:0}: Error finding container 0ad8adbaf6b4ff768891c74f1a1809de290c4343913178719e7db009540ff271: Status 404 returned error can't find the container with id 0ad8adbaf6b4ff768891c74f1a1809de290c4343913178719e7db009540ff271 Oct 31 19:56:09.592762 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:09.592716 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" podStartSLOduration=1.592700257 podStartE2EDuration="1.592700257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:09.59115067 +0000 UTC m=+2341.224544577" watchObservedRunningTime="2026-10-31 19:56:09.592700257 +0000 UTC m=+2341.226094165" Oct 31 19:56:10.489022 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:10.488993 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" Oct 31 19:56:10.489178 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:10.489040 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" event={"ID":"f7d03485-903b-480b-a923-08a9fd632e7b","Type":"ContainerStarted","Data":"5cbafebe41170b87999632cba882bcbfc27b5c1a00f53b5f73996394913b71ba"} Oct 31 19:56:10.489178 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:10.489061 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" event={"ID":"f7d03485-903b-480b-a923-08a9fd632e7b","Type":"ContainerStarted","Data":"0ad8adbaf6b4ff768891c74f1a1809de290c4343913178719e7db009540ff271"} Oct 31 19:56:10.595472 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:10.595420 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" podStartSLOduration=2.595407008 podStartE2EDuration="2.595407008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:10.59366057 +0000 UTC m=+2342.227054479" watchObservedRunningTime="2026-10-31 19:56:10.595407008 +0000 UTC m=+2342.228800917" Oct 31 19:56:20.406038 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:56:20.405952 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:56:20.490061 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:20.490032 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" Oct 31 19:56:21.497642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:21.497612 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" Oct 31 19:56:50.405619 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:56:50.405562 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:56:52.731933 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:52.731898 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96"] Oct 31 19:56:52.732376 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:52.732124 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" containerID="cri-o://5bb6a5a8f766ab851ff05ba8536e5366f21ba7f5f2b8573989f0cbf545c5314f" gracePeriod=30 Oct 31 19:56:55.566507 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.566332 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4"] Oct 31 19:56:55.601342 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.588865 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" Oct 31 19:56:55.641911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.639749 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4\" (UID: \"0ec282e8-0fc4-4bdb-9212-9ab568c6200f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" Oct 31 19:56:55.641911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.639986 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr9lt\" (UniqueName: \"kubernetes.io/projected/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-kube-api-access-cr9lt\") pod \"payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4\" (UID: \"0ec282e8-0fc4-4bdb-9212-9ab568c6200f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" Oct 31 19:56:55.660905 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.651217 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-ddb5a8\"" Oct 31 19:56:55.682913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.672278 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-ddb5a8-dockercfg-kkzkg\"" Oct 31 19:56:55.682913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.675205 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4"] Oct 31 19:56:55.751418 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.751365 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx"] Oct 31 19:56:55.758851 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.755675 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4\" (UID: \"0ec282e8-0fc4-4bdb-9212-9ab568c6200f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" Oct 31 19:56:55.759328 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.759286 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr9lt\" (UniqueName: \"kubernetes.io/projected/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-kube-api-access-cr9lt\") pod \"payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4\" (UID: \"0ec282e8-0fc4-4bdb-9212-9ab568c6200f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" Oct 31 19:56:55.760009 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.756519 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4\" (UID: \"0ec282e8-0fc4-4bdb-9212-9ab568c6200f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" Oct 31 19:56:55.765672 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.765638 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" Oct 31 19:56:55.775771 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.775633 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx"] Oct 31 19:56:55.814598 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.814565 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr9lt\" (UniqueName: \"kubernetes.io/projected/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-kube-api-access-cr9lt\") pod \"payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4\" (UID: \"0ec282e8-0fc4-4bdb-9212-9ab568c6200f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" Oct 31 19:56:55.860800 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.860709 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cv8q\" (UniqueName: \"kubernetes.io/projected/fda177cc-c3ca-42a4-9104-6da967a51a4f-kube-api-access-9cv8q\") pod \"payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx\" (UID: \"fda177cc-c3ca-42a4-9104-6da967a51a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" Oct 31 19:56:55.861084 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.860958 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda177cc-c3ca-42a4-9104-6da967a51a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx\" (UID: \"fda177cc-c3ca-42a4-9104-6da967a51a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" Oct 31 19:56:55.955839 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.955800 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" Oct 31 19:56:55.961971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.961938 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cv8q\" (UniqueName: \"kubernetes.io/projected/fda177cc-c3ca-42a4-9104-6da967a51a4f-kube-api-access-9cv8q\") pod \"payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx\" (UID: \"fda177cc-c3ca-42a4-9104-6da967a51a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" Oct 31 19:56:55.962179 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.962164 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda177cc-c3ca-42a4-9104-6da967a51a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx\" (UID: \"fda177cc-c3ca-42a4-9104-6da967a51a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" Oct 31 19:56:55.963056 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:55.963029 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda177cc-c3ca-42a4-9104-6da967a51a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx\" (UID: \"fda177cc-c3ca-42a4-9104-6da967a51a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" Oct 31 19:56:56.029055 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.023755 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cv8q\" (UniqueName: \"kubernetes.io/projected/fda177cc-c3ca-42a4-9104-6da967a51a4f-kube-api-access-9cv8q\") pod \"payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx\" (UID: \"fda177cc-c3ca-42a4-9104-6da967a51a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" Oct 31 19:56:56.029055 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.024161 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx"] Oct 31 19:56:56.033145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.032097 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.063615 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.063524 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-socket\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.063615 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.063593 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-podinfo\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.063845 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.063718 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-credential-socket\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.063845 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.063838 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l97qp\" (UniqueName: \"kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-kube-api-access-l97qp\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.063981 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.063867 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-envoy\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.063981 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.063936 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istiod-ca-cert\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.064151 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.064104 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-certs\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.064151 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.064142 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-token\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.064255 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.064216 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-data\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.084510 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.084009 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx"] Oct 31 19:56:56.085357 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.085334 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9bbcf7-openshift-default-dockercfg-9dbjl\"" Oct 31 19:56:56.108453 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.108248 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" Oct 31 19:56:56.168395 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.167968 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-podinfo\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.168395 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.168049 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-credential-socket\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.168395 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.168142 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l97qp\" (UniqueName: \"kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-kube-api-access-l97qp\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.168395 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.168175 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-envoy\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.168395 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.168199 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istiod-ca-cert\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.168395 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.168322 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-certs\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.168395 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.168353 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-token\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.168930 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.168422 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-data\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.168930 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.168503 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-socket\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.169110 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.169082 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-socket\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.169423 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.169407 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-credential-socket\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.171856 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.171827 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-certs\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.171986 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.171915 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istiod-ca-cert\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.172057 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.172032 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-data\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.172466 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.172441 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-podinfo\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.173149 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.173130 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-envoy\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.245427 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.245391 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l97qp\" (UniqueName: \"kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-kube-api-access-l97qp\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.247944 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.247827 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-token\") pod \"e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.358967 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.358932 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4"] Oct 31 19:56:56.360808 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:56:56.360773 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec282e8_0fc4_4bdb_9212_9ab568c6200f.slice/crio-64abebc0c83316126045276816f47aea3b84654e0db3972d939ebc2872da0445 WatchSource:0}: Error finding container 64abebc0c83316126045276816f47aea3b84654e0db3972d939ebc2872da0445: Status 404 returned error can't find the container with id 64abebc0c83316126045276816f47aea3b84654e0db3972d939ebc2872da0445 Oct 31 19:56:56.362260 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.362197 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:56.557538 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.557508 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx"] Oct 31 19:56:56.558056 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:56:56.558028 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda177cc_c3ca_42a4_9104_6da967a51a4f.slice/crio-738211893a216e96299cc50631d3f460488253a212fcf41162ba94f59ff5198e WatchSource:0}: Error finding container 738211893a216e96299cc50631d3f460488253a212fcf41162ba94f59ff5198e: Status 404 returned error can't find the container with id 738211893a216e96299cc50631d3f460488253a212fcf41162ba94f59ff5198e Oct 31 19:56:56.754904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.750998 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx"] Oct 31 19:56:56.757604 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.757563 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:56:56.757710 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.757641 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:56:56.757710 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.757672 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:56:56.877642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.877531 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" event={"ID":"63dee691-27c6-4a5f-882c-fc3ed46cbeb1","Type":"ContainerStarted","Data":"aeb1ffc08c610db47e0d29ac08780c55190d5f1a65d0bc52e67af92521a5c81c"} Oct 31 19:56:56.877642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.877593 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" event={"ID":"63dee691-27c6-4a5f-882c-fc3ed46cbeb1","Type":"ContainerStarted","Data":"3fcae3a00e22db7bd973db1a7d82778f95f7eb746fd90a869279861c0bcb7cd0"} Oct 31 19:56:56.879642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.879608 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" event={"ID":"0ec282e8-0fc4-4bdb-9212-9ab568c6200f","Type":"ContainerStarted","Data":"30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4"} Oct 31 19:56:56.879793 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.879666 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" event={"ID":"0ec282e8-0fc4-4bdb-9212-9ab568c6200f","Type":"ContainerStarted","Data":"64abebc0c83316126045276816f47aea3b84654e0db3972d939ebc2872da0445"} Oct 31 19:56:56.879793 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.879700 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" Oct 31 19:56:56.881830 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.881806 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" event={"ID":"fda177cc-c3ca-42a4-9104-6da967a51a4f","Type":"ContainerStarted","Data":"cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11"} Oct 31 19:56:56.881972 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.881845 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" event={"ID":"fda177cc-c3ca-42a4-9104-6da967a51a4f","Type":"ContainerStarted","Data":"738211893a216e96299cc50631d3f460488253a212fcf41162ba94f59ff5198e"} Oct 31 19:56:56.881972 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.881930 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" Oct 31 19:56:56.992370 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:56.992317 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" podStartSLOduration=1.9923019979999999 podStartE2EDuration="1.992301998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:56.989085804 +0000 UTC m=+2388.622479712" watchObservedRunningTime="2026-10-31 19:56:56.992301998 +0000 UTC m=+2388.625695960" Oct 31 19:56:57.097309 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:57.097257 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" podStartSLOduration=2.097243451 podStartE2EDuration="2.097243451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:57.093774102 +0000 UTC m=+2388.727168011" watchObservedRunningTime="2026-10-31 19:56:57.097243451 +0000 UTC m=+2388.730637360" Oct 31 19:56:57.210307 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:57.210211 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" podStartSLOduration=2.210196268 podStartE2EDuration="2.210196268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:57.205208744 +0000 UTC m=+2388.838602654" watchObservedRunningTime="2026-10-31 19:56:57.210196268 +0000 UTC m=+2388.843590176" Oct 31 19:56:57.363059 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:57.363025 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:57.370456 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:57.370426 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:57.893914 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:57.892236 6107 generic.go:356] "Generic (PLEG): container finished" podID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerID="5bb6a5a8f766ab851ff05ba8536e5366f21ba7f5f2b8573989f0cbf545c5314f" exitCode=0 Oct 31 19:56:57.893914 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:57.892313 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" event={"ID":"3468d78e-548e-41b5-a285-73c62cbe20cc","Type":"ContainerDied","Data":"5bb6a5a8f766ab851ff05ba8536e5366f21ba7f5f2b8573989f0cbf545c5314f"} Oct 31 19:56:57.893914 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:57.893144 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:57.894493 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:57.894104 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:56:58.054994 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.054954 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:56:58.097525 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.097479 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-envoy\") pod \"3468d78e-548e-41b5-a285-73c62cbe20cc\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " Oct 31 19:56:58.097731 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.097551 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3468d78e-548e-41b5-a285-73c62cbe20cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3468d78e-548e-41b5-a285-73c62cbe20cc-istiod-ca-cert\") pod \"3468d78e-548e-41b5-a285-73c62cbe20cc\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " Oct 31 19:56:58.097731 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.097670 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-kube-api-access-p494r\" (UniqueName: \"kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-kube-api-access-p494r\") pod \"3468d78e-548e-41b5-a285-73c62cbe20cc\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " Oct 31 19:56:58.097731 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.097725 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-podinfo\") pod \"3468d78e-548e-41b5-a285-73c62cbe20cc\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " Oct 31 19:56:58.097933 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.097752 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-credential-socket\") pod \"3468d78e-548e-41b5-a285-73c62cbe20cc\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " Oct 31 19:56:58.097933 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.097837 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-socket\") pod \"3468d78e-548e-41b5-a285-73c62cbe20cc\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " Oct 31 19:56:58.098040 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.097976 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-data\") pod \"3468d78e-548e-41b5-a285-73c62cbe20cc\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " Oct 31 19:56:58.098096 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.098069 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-token\" (UniqueName: \"kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-token\") pod \"3468d78e-548e-41b5-a285-73c62cbe20cc\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " Oct 31 19:56:58.098096 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.098067 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-credential-socket" pod "3468d78e-548e-41b5-a285-73c62cbe20cc" (UID: "3468d78e-548e-41b5-a285-73c62cbe20cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:58.098211 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.098086 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-socket" pod "3468d78e-548e-41b5-a285-73c62cbe20cc" (UID: "3468d78e-548e-41b5-a285-73c62cbe20cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:58.098211 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.098170 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-certs\") pod \"3468d78e-548e-41b5-a285-73c62cbe20cc\" (UID: \"3468d78e-548e-41b5-a285-73c62cbe20cc\") " Oct 31 19:56:58.098313 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.098167 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3468d78e-548e-41b5-a285-73c62cbe20cc-istiod-ca-cert" pod "3468d78e-548e-41b5-a285-73c62cbe20cc" (UID: "3468d78e-548e-41b5-a285-73c62cbe20cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:58.098313 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.098244 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-data" pod "3468d78e-548e-41b5-a285-73c62cbe20cc" (UID: "3468d78e-548e-41b5-a285-73c62cbe20cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:58.098468 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.098449 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-certs" pod "3468d78e-548e-41b5-a285-73c62cbe20cc" (UID: "3468d78e-548e-41b5-a285-73c62cbe20cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:58.098819 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.098799 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.098819 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.098818 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.098980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.098827 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.098980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.098837 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3468d78e-548e-41b5-a285-73c62cbe20cc-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.098980 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.098846 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.099932 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.099857 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-podinfo" pod "3468d78e-548e-41b5-a285-73c62cbe20cc" (UID: "3468d78e-548e-41b5-a285-73c62cbe20cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:56:58.100419 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.100396 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-envoy" pod "3468d78e-548e-41b5-a285-73c62cbe20cc" (UID: "3468d78e-548e-41b5-a285-73c62cbe20cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:58.100506 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.100453 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-token" pod "3468d78e-548e-41b5-a285-73c62cbe20cc" (UID: "3468d78e-548e-41b5-a285-73c62cbe20cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:58.100506 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.100485 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-kube-api-access-p494r" pod "3468d78e-548e-41b5-a285-73c62cbe20cc" (UID: "3468d78e-548e-41b5-a285-73c62cbe20cc"). InnerVolumeSpecName "kube-api-access-p494r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:58.201690 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.201656 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.201946 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.201929 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.202042 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.202028 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p494r\" (UniqueName: \"kubernetes.io/projected/3468d78e-548e-41b5-a285-73c62cbe20cc-kube-api-access-p494r\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.202152 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.202120 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3468d78e-548e-41b5-a285-73c62cbe20cc-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.912931 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.912830 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" Oct 31 19:56:58.912931 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.912855 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96" event={"ID":"3468d78e-548e-41b5-a285-73c62cbe20cc","Type":"ContainerDied","Data":"75e6de91ae44db73d7c39d0e264eff4691839e2a2c46e936e49640cbf91fad9c"} Oct 31 19:56:58.913510 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:58.912941 6107 scope.go:122] "RemoveContainer" containerID="5bb6a5a8f766ab851ff05ba8536e5366f21ba7f5f2b8573989f0cbf545c5314f" Oct 31 19:56:59.068518 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:59.068477 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96"] Oct 31 19:56:59.116409 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:56:59.116359 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-rbk96"] Oct 31 19:57:00.488701 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:00.488666 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" path="/var/lib/kubelet/pods/3468d78e-548e-41b5-a285-73c62cbe20cc/volumes" Oct 31 19:57:04.462666 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.462609 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc"] Oct 31 19:57:04.463753 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.463727 6107 cpu_manager.go:414] "containerMap: removing container" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" Oct 31 19:57:04.464340 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.463754 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" Oct 31 19:57:04.464340 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.464008 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="3468d78e-548e-41b5-a285-73c62cbe20cc" containerName="istio-proxy" Oct 31 19:57:04.471165 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.471138 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" Oct 31 19:57:04.522605 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.520596 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc"] Oct 31 19:57:04.549002 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.548960 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-888lj\" (UniqueName: \"kubernetes.io/projected/81d92768-7aee-4fa2-9777-d13bcc073772-kube-api-access-888lj\") pod \"payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc\" (UID: \"81d92768-7aee-4fa2-9777-d13bcc073772\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" Oct 31 19:57:04.549287 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.549244 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d92768-7aee-4fa2-9777-d13bcc073772-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc\" (UID: \"81d92768-7aee-4fa2-9777-d13bcc073772\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" Oct 31 19:57:04.561263 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.561225 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9bbcf7\"" Oct 31 19:57:04.561472 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.561320 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9bbcf7-dockercfg-g7lvf\"" Oct 31 19:57:04.622904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.621979 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb"] Oct 31 19:57:04.652170 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.643487 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" Oct 31 19:57:04.666213 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.661215 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-888lj\" (UniqueName: \"kubernetes.io/projected/81d92768-7aee-4fa2-9777-d13bcc073772-kube-api-access-888lj\") pod \"payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc\" (UID: \"81d92768-7aee-4fa2-9777-d13bcc073772\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" Oct 31 19:57:04.673092 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.673057 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d92768-7aee-4fa2-9777-d13bcc073772-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc\" (UID: \"81d92768-7aee-4fa2-9777-d13bcc073772\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" Oct 31 19:57:04.675741 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.673942 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d92768-7aee-4fa2-9777-d13bcc073772-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc\" (UID: \"81d92768-7aee-4fa2-9777-d13bcc073772\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" Oct 31 19:57:04.691175 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.689788 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb"] Oct 31 19:57:04.723207 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.722006 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-888lj\" (UniqueName: \"kubernetes.io/projected/81d92768-7aee-4fa2-9777-d13bcc073772-kube-api-access-888lj\") pod \"payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc\" (UID: \"81d92768-7aee-4fa2-9777-d13bcc073772\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" Oct 31 19:57:04.776902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.775969 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb\" (UID: \"75491f8d-9f81-4f0b-afbf-d2a63b7b6731\") " pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" Oct 31 19:57:04.776902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.776045 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9m5\" (UniqueName: \"kubernetes.io/projected/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-kube-api-access-tk9m5\") pod \"payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb\" (UID: \"75491f8d-9f81-4f0b-afbf-d2a63b7b6731\") " pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" Oct 31 19:57:04.791815 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.791783 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" Oct 31 19:57:04.883205 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.883163 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb\" (UID: \"75491f8d-9f81-4f0b-afbf-d2a63b7b6731\") " pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" Oct 31 19:57:04.883364 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.883250 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9m5\" (UniqueName: \"kubernetes.io/projected/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-kube-api-access-tk9m5\") pod \"payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb\" (UID: \"75491f8d-9f81-4f0b-afbf-d2a63b7b6731\") " pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" Oct 31 19:57:04.884732 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.884704 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb\" (UID: \"75491f8d-9f81-4f0b-afbf-d2a63b7b6731\") " pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" Oct 31 19:57:04.937480 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.937444 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9m5\" (UniqueName: \"kubernetes.io/projected/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-kube-api-access-tk9m5\") pod \"payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb\" (UID: \"75491f8d-9f81-4f0b-afbf-d2a63b7b6731\") " pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" Oct 31 19:57:04.999040 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:04.998948 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" Oct 31 19:57:05.173144 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:05.173111 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc"] Oct 31 19:57:05.173709 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:57:05.173668 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d92768_7aee_4fa2_9777_d13bcc073772.slice/crio-ab22d5e445186264e8641d3adf2c6633325c67e0ba92e5d108bed61b17d02219 WatchSource:0}: Error finding container ab22d5e445186264e8641d3adf2c6633325c67e0ba92e5d108bed61b17d02219: Status 404 returned error can't find the container with id ab22d5e445186264e8641d3adf2c6633325c67e0ba92e5d108bed61b17d02219 Oct 31 19:57:05.381497 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:05.381448 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb"] Oct 31 19:57:05.381909 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:57:05.381847 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75491f8d_9f81_4f0b_afbf_d2a63b7b6731.slice/crio-cf2857eab0b5311994a155b83658154ea4c3433fdc018d373ea3632a1758d997 WatchSource:0}: Error finding container cf2857eab0b5311994a155b83658154ea4c3433fdc018d373ea3632a1758d997: Status 404 returned error can't find the container with id cf2857eab0b5311994a155b83658154ea4c3433fdc018d373ea3632a1758d997 Oct 31 19:57:05.979108 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:05.979076 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" event={"ID":"75491f8d-9f81-4f0b-afbf-d2a63b7b6731","Type":"ContainerStarted","Data":"275ff3fc0b6d002f89a3290056cf77343238cde248030626d1c10eee32c87a1e"} Oct 31 19:57:05.979562 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:05.979132 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" event={"ID":"75491f8d-9f81-4f0b-afbf-d2a63b7b6731","Type":"ContainerStarted","Data":"cf2857eab0b5311994a155b83658154ea4c3433fdc018d373ea3632a1758d997"} Oct 31 19:57:05.979562 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:05.979167 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" Oct 31 19:57:05.981422 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:05.981394 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" event={"ID":"81d92768-7aee-4fa2-9777-d13bcc073772","Type":"ContainerStarted","Data":"cdfdd120eae2104565a093f18475544c6b7dff9035ec7a73abbbadfc3aead4cd"} Oct 31 19:57:05.981559 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:05.981440 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" event={"ID":"81d92768-7aee-4fa2-9777-d13bcc073772","Type":"ContainerStarted","Data":"ab22d5e445186264e8641d3adf2c6633325c67e0ba92e5d108bed61b17d02219"} Oct 31 19:57:05.981559 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:05.981463 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" Oct 31 19:57:06.204201 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:06.204148 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" podStartSLOduration=2.204133153 podStartE2EDuration="2.204133153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:06.096095805 +0000 UTC m=+2397.729489716" watchObservedRunningTime="2026-10-31 19:57:06.204133153 +0000 UTC m=+2397.837527097" Oct 31 19:57:06.205440 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:06.205127 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" podStartSLOduration=2.205117371 podStartE2EDuration="2.205117371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:06.201117803 +0000 UTC m=+2397.834511712" watchObservedRunningTime="2026-10-31 19:57:06.205117371 +0000 UTC m=+2397.838511280" Oct 31 19:57:07.896428 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:07.896401 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" Oct 31 19:57:07.896841 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:07.896469 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" Oct 31 19:57:08.281413 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:08.281379 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh"] Oct 31 19:57:08.281616 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:08.281585 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" podUID="4d4507c2-4259-4de9-91d6-49329d702cca" containerName="istio-proxy" containerID="cri-o://d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026" gracePeriod=30 Oct 31 19:57:09.233863 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:09.233829 6107 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:57:09.234318 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:09.233900 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" podUID="4d4507c2-4259-4de9-91d6-49329d702cca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:57:13.565225 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.565169 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:57:13.630230 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630197 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-envoy\") pod \"4d4507c2-4259-4de9-91d6-49329d702cca\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " Oct 31 19:57:13.630435 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630308 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-socket\") pod \"4d4507c2-4259-4de9-91d6-49329d702cca\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " Oct 31 19:57:13.630435 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630346 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d4507c2-4259-4de9-91d6-49329d702cca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4507c2-4259-4de9-91d6-49329d702cca-istiod-ca-cert\") pod \"4d4507c2-4259-4de9-91d6-49329d702cca\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " Oct 31 19:57:13.630545 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630475 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-certs\") pod \"4d4507c2-4259-4de9-91d6-49329d702cca\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " Oct 31 19:57:13.630545 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630509 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-data\") pod \"4d4507c2-4259-4de9-91d6-49329d702cca\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " Oct 31 19:57:13.630644 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630585 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d4507c2-4259-4de9-91d6-49329d702cca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4507c2-4259-4de9-91d6-49329d702cca-istio-podinfo\") pod \"4d4507c2-4259-4de9-91d6-49329d702cca\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " Oct 31 19:57:13.630700 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630636 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-socket" pod "4d4507c2-4259-4de9-91d6-49329d702cca" (UID: "4d4507c2-4259-4de9-91d6-49329d702cca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:57:13.630754 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630732 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-kube-api-access-zjcnw\" (UniqueName: \"kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-kube-api-access-zjcnw\") pod \"4d4507c2-4259-4de9-91d6-49329d702cca\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " Oct 31 19:57:13.630804 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630745 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4507c2-4259-4de9-91d6-49329d702cca-istiod-ca-cert" pod "4d4507c2-4259-4de9-91d6-49329d702cca" (UID: "4d4507c2-4259-4de9-91d6-49329d702cca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:57:13.630804 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630786 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-credential-socket\") pod \"4d4507c2-4259-4de9-91d6-49329d702cca\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " Oct 31 19:57:13.630921 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630829 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-istio-token\" (UniqueName: \"kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-istio-token\") pod \"4d4507c2-4259-4de9-91d6-49329d702cca\" (UID: \"4d4507c2-4259-4de9-91d6-49329d702cca\") " Oct 31 19:57:13.630921 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630840 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-data" pod "4d4507c2-4259-4de9-91d6-49329d702cca" (UID: "4d4507c2-4259-4de9-91d6-49329d702cca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:57:13.630921 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.630860 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-certs" pod "4d4507c2-4259-4de9-91d6-49329d702cca" (UID: "4d4507c2-4259-4de9-91d6-49329d702cca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:57:13.631150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.631129 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-credential-socket" pod "4d4507c2-4259-4de9-91d6-49329d702cca" (UID: "4d4507c2-4259-4de9-91d6-49329d702cca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:57:13.631532 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.631505 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:13.631532 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.631534 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:13.631771 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.631550 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4507c2-4259-4de9-91d6-49329d702cca-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:13.631771 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.631566 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:13.631771 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.631581 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:13.633285 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.633258 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-envoy" pod "4d4507c2-4259-4de9-91d6-49329d702cca" (UID: "4d4507c2-4259-4de9-91d6-49329d702cca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:57:13.633285 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.633276 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d4507c2-4259-4de9-91d6-49329d702cca-istio-podinfo" pod "4d4507c2-4259-4de9-91d6-49329d702cca" (UID: "4d4507c2-4259-4de9-91d6-49329d702cca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:57:13.633597 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.633569 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-istio-token" pod "4d4507c2-4259-4de9-91d6-49329d702cca" (UID: "4d4507c2-4259-4de9-91d6-49329d702cca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:57:13.634726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.634703 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-kube-api-access-zjcnw" pod "4d4507c2-4259-4de9-91d6-49329d702cca" (UID: "4d4507c2-4259-4de9-91d6-49329d702cca"). InnerVolumeSpecName "kube-api-access-zjcnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:57:13.733343 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.733308 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4507c2-4259-4de9-91d6-49329d702cca-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:13.733343 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.733341 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjcnw\" (UniqueName: \"kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-kube-api-access-zjcnw\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:13.733549 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.733354 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4507c2-4259-4de9-91d6-49329d702cca-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:13.733549 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:13.733368 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4507c2-4259-4de9-91d6-49329d702cca-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:14.056328 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:14.056289 6107 generic.go:356] "Generic (PLEG): container finished" podID="4d4507c2-4259-4de9-91d6-49329d702cca" containerID="d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026" exitCode=0 Oct 31 19:57:14.056513 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:14.056342 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" Oct 31 19:57:14.056513 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:14.056369 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" event={"ID":"4d4507c2-4259-4de9-91d6-49329d702cca","Type":"ContainerDied","Data":"d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026"} Oct 31 19:57:14.056513 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:14.056428 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh" event={"ID":"4d4507c2-4259-4de9-91d6-49329d702cca","Type":"ContainerDied","Data":"dd20675fc864f3e27d99265324efc49806ed81d0064113644b22a48702ac718f"} Oct 31 19:57:14.056513 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:14.056451 6107 scope.go:122] "RemoveContainer" containerID="d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026" Oct 31 19:57:14.068093 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:14.068074 6107 scope.go:122] "RemoveContainer" containerID="d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026" Oct 31 19:57:14.068365 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:57:14.068345 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026\": container with ID starting with d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026 not found: ID does not exist" containerID="d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026" Oct 31 19:57:14.068426 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:14.068375 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026"} err="failed to get container status \"d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026\": rpc error: code = NotFound desc = could not find container \"d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026\": container with ID starting with d3b68bb66b3eb53f08d934dfdee4386665588501f83090e28dbfd9e36ed4f026 not found: ID does not exist" Oct 31 19:57:14.210714 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:14.210680 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh"] Oct 31 19:57:14.256312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:14.256281 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-xwvbh"] Oct 31 19:57:14.488993 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:14.488955 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d4507c2-4259-4de9-91d6-49329d702cca" path="/var/lib/kubelet/pods/4d4507c2-4259-4de9-91d6-49329d702cca/volumes" Oct 31 19:57:15.054799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.052013 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz"] Oct 31 19:57:15.054799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.053175 6107 cpu_manager.go:414] "containerMap: removing container" podUID="4d4507c2-4259-4de9-91d6-49329d702cca" containerName="istio-proxy" Oct 31 19:57:15.054799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.053194 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4507c2-4259-4de9-91d6-49329d702cca" containerName="istio-proxy" Oct 31 19:57:15.054799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.053354 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4507c2-4259-4de9-91d6-49329d702cca" containerName="istio-proxy" Oct 31 19:57:15.061006 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.060979 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" Oct 31 19:57:15.102671 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.102641 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz"] Oct 31 19:57:15.110058 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.110030 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-650cf3\"" Oct 31 19:57:15.110615 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.110594 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-650cf3-dockercfg-mxqwb\"" Oct 31 19:57:15.276906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.264038 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8bh\" (UniqueName: \"kubernetes.io/projected/d9267cfb-f5bf-4b85-bfef-9327801c261b-kube-api-access-zc8bh\") pod \"payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz\" (UID: \"d9267cfb-f5bf-4b85-bfef-9327801c261b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" Oct 31 19:57:15.276906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.264223 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9267cfb-f5bf-4b85-bfef-9327801c261b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz\" (UID: \"d9267cfb-f5bf-4b85-bfef-9327801c261b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" Oct 31 19:57:15.288907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.279552 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7"] Oct 31 19:57:15.288907 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.286338 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" Oct 31 19:57:15.318426 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.316587 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7"] Oct 31 19:57:15.365817 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.365783 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fnlw\" (UniqueName: \"kubernetes.io/projected/3b785406-50fa-48e4-aa62-d36c95495ca4-kube-api-access-6fnlw\") pod \"payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7\" (UID: \"3b785406-50fa-48e4-aa62-d36c95495ca4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" Oct 31 19:57:15.366013 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.365860 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8bh\" (UniqueName: \"kubernetes.io/projected/d9267cfb-f5bf-4b85-bfef-9327801c261b-kube-api-access-zc8bh\") pod \"payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz\" (UID: \"d9267cfb-f5bf-4b85-bfef-9327801c261b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" Oct 31 19:57:15.366127 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.366099 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9267cfb-f5bf-4b85-bfef-9327801c261b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz\" (UID: \"d9267cfb-f5bf-4b85-bfef-9327801c261b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" Oct 31 19:57:15.366184 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.366169 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b785406-50fa-48e4-aa62-d36c95495ca4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7\" (UID: \"3b785406-50fa-48e4-aa62-d36c95495ca4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" Oct 31 19:57:15.366850 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.366824 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9267cfb-f5bf-4b85-bfef-9327801c261b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz\" (UID: \"d9267cfb-f5bf-4b85-bfef-9327801c261b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" Oct 31 19:57:15.418478 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.418448 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8bh\" (UniqueName: \"kubernetes.io/projected/d9267cfb-f5bf-4b85-bfef-9327801c261b-kube-api-access-zc8bh\") pod \"payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz\" (UID: \"d9267cfb-f5bf-4b85-bfef-9327801c261b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" Oct 31 19:57:15.467549 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.467512 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b785406-50fa-48e4-aa62-d36c95495ca4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7\" (UID: \"3b785406-50fa-48e4-aa62-d36c95495ca4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" Oct 31 19:57:15.467705 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.467627 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fnlw\" (UniqueName: \"kubernetes.io/projected/3b785406-50fa-48e4-aa62-d36c95495ca4-kube-api-access-6fnlw\") pod \"payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7\" (UID: \"3b785406-50fa-48e4-aa62-d36c95495ca4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" Oct 31 19:57:15.468251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.468230 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b785406-50fa-48e4-aa62-d36c95495ca4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7\" (UID: \"3b785406-50fa-48e4-aa62-d36c95495ca4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" Oct 31 19:57:15.520286 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.520261 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fnlw\" (UniqueName: \"kubernetes.io/projected/3b785406-50fa-48e4-aa62-d36c95495ca4-kube-api-access-6fnlw\") pod \"payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7\" (UID: \"3b785406-50fa-48e4-aa62-d36c95495ca4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" Oct 31 19:57:15.625170 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.625008 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" Oct 31 19:57:15.677901 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:15.676894 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" Oct 31 19:57:16.000437 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:16.000407 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7"] Oct 31 19:57:16.001564 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:57:16.001535 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b785406_50fa_48e4_aa62_d36c95495ca4.slice/crio-58a155c904df8b1a386fbe5f846a861d611e4365ad71ca3f2cf41bb193cb63b3 WatchSource:0}: Error finding container 58a155c904df8b1a386fbe5f846a861d611e4365ad71ca3f2cf41bb193cb63b3: Status 404 returned error can't find the container with id 58a155c904df8b1a386fbe5f846a861d611e4365ad71ca3f2cf41bb193cb63b3 Oct 31 19:57:16.076663 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:16.076634 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" event={"ID":"3b785406-50fa-48e4-aa62-d36c95495ca4","Type":"ContainerStarted","Data":"58a155c904df8b1a386fbe5f846a861d611e4365ad71ca3f2cf41bb193cb63b3"} Oct 31 19:57:16.095190 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:16.095161 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz"] Oct 31 19:57:16.095606 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:57:16.095578 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9267cfb_f5bf_4b85_bfef_9327801c261b.slice/crio-5b7b8f3d36e4a2c987d3a8127f8f78d897fde5b2eed0e056a476d51baf473837 WatchSource:0}: Error finding container 5b7b8f3d36e4a2c987d3a8127f8f78d897fde5b2eed0e056a476d51baf473837: Status 404 returned error can't find the container with id 5b7b8f3d36e4a2c987d3a8127f8f78d897fde5b2eed0e056a476d51baf473837 Oct 31 19:57:16.996800 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:16.996762 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" Oct 31 19:57:16.997006 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:16.996856 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" Oct 31 19:57:17.085320 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:17.085285 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" event={"ID":"3b785406-50fa-48e4-aa62-d36c95495ca4","Type":"ContainerStarted","Data":"a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3"} Oct 31 19:57:17.085800 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:17.085367 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" Oct 31 19:57:17.086871 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:17.086851 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" event={"ID":"d9267cfb-f5bf-4b85-bfef-9327801c261b","Type":"ContainerStarted","Data":"d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32"} Oct 31 19:57:17.086998 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:17.086899 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" event={"ID":"d9267cfb-f5bf-4b85-bfef-9327801c261b","Type":"ContainerStarted","Data":"5b7b8f3d36e4a2c987d3a8127f8f78d897fde5b2eed0e056a476d51baf473837"} Oct 31 19:57:17.086998 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:17.086987 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" Oct 31 19:57:17.318532 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:17.318416 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" podStartSLOduration=2.318398889 podStartE2EDuration="2.318398889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:17.318096624 +0000 UTC m=+2408.951490534" watchObservedRunningTime="2026-10-31 19:57:17.318398889 +0000 UTC m=+2408.951792806" Oct 31 19:57:17.435614 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:17.425281 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" podStartSLOduration=2.425266218 podStartE2EDuration="2.425266218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:17.422125501 +0000 UTC m=+2409.055519436" watchObservedRunningTime="2026-10-31 19:57:17.425266218 +0000 UTC m=+2409.058660128" Oct 31 19:57:20.405075 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:57:20.405036 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:57:28.098198 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:28.098166 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" Oct 31 19:57:28.098610 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:57:28.098258 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" Oct 31 19:57:50.405996 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:57:50.405793 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:58:09.225801 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:09.225760 6107 scope.go:122] "RemoveContainer" containerID="911d30dcf7dcb6b265cb5360deccca12f7c4f2c8578587b05e40ab71d2982d78" Oct 31 19:58:10.475721 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.475682 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk"] Oct 31 19:58:10.482629 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.482596 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" Oct 31 19:58:10.601349 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.601276 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dfe658-dockercfg-zw2l4\"" Oct 31 19:58:10.601753 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.601736 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dfe658\"" Oct 31 19:58:10.607952 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.603391 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk\" (UID: \"f0c12fb7-e4fd-40f1-8a3d-6909084f0394\") " pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" Oct 31 19:58:10.631901 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.608428 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgdzk\" (UniqueName: \"kubernetes.io/projected/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-kube-api-access-vgdzk\") pod \"payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk\" (UID: \"f0c12fb7-e4fd-40f1-8a3d-6909084f0394\") " pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" Oct 31 19:58:10.671534 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.671500 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk"] Oct 31 19:58:10.679199 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.679169 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm"] Oct 31 19:58:10.695794 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.695757 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" Oct 31 19:58:10.716315 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.716280 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm"] Oct 31 19:58:10.765860 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.765767 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6949x\" (UniqueName: \"kubernetes.io/projected/4b991a69-1862-41cf-b122-6bb782044784-kube-api-access-6949x\") pod \"payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm\" (UID: \"4b991a69-1862-41cf-b122-6bb782044784\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" Oct 31 19:58:10.766076 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.765941 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk\" (UID: \"f0c12fb7-e4fd-40f1-8a3d-6909084f0394\") " pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" Oct 31 19:58:10.766148 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.766073 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdzk\" (UniqueName: \"kubernetes.io/projected/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-kube-api-access-vgdzk\") pod \"payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk\" (UID: \"f0c12fb7-e4fd-40f1-8a3d-6909084f0394\") " pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" Oct 31 19:58:10.766207 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.766142 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b991a69-1862-41cf-b122-6bb782044784-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm\" (UID: \"4b991a69-1862-41cf-b122-6bb782044784\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" Oct 31 19:58:10.766690 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.766663 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk\" (UID: \"f0c12fb7-e4fd-40f1-8a3d-6909084f0394\") " pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" Oct 31 19:58:10.818718 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.818689 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdzk\" (UniqueName: \"kubernetes.io/projected/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-kube-api-access-vgdzk\") pod \"payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk\" (UID: \"f0c12fb7-e4fd-40f1-8a3d-6909084f0394\") " pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" Oct 31 19:58:10.824649 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.824621 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" Oct 31 19:58:10.867158 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.867128 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b991a69-1862-41cf-b122-6bb782044784-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm\" (UID: \"4b991a69-1862-41cf-b122-6bb782044784\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" Oct 31 19:58:10.867345 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.867236 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6949x\" (UniqueName: \"kubernetes.io/projected/4b991a69-1862-41cf-b122-6bb782044784-kube-api-access-6949x\") pod \"payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm\" (UID: \"4b991a69-1862-41cf-b122-6bb782044784\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" Oct 31 19:58:10.868075 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.868048 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b991a69-1862-41cf-b122-6bb782044784-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm\" (UID: \"4b991a69-1862-41cf-b122-6bb782044784\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" Oct 31 19:58:10.920316 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:10.920283 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6949x\" (UniqueName: \"kubernetes.io/projected/4b991a69-1862-41cf-b122-6bb782044784-kube-api-access-6949x\") pod \"payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm\" (UID: \"4b991a69-1862-41cf-b122-6bb782044784\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" Oct 31 19:58:11.018102 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:11.018014 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" Oct 31 19:58:11.204447 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:11.204414 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk"] Oct 31 19:58:11.204619 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:58:11.204596 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c12fb7_e4fd_40f1_8a3d_6909084f0394.slice/crio-b724c156c701f6e470bd342c365a9276a0a2f122f1b3ea10e98b2dce91f2f017 WatchSource:0}: Error finding container b724c156c701f6e470bd342c365a9276a0a2f122f1b3ea10e98b2dce91f2f017: Status 404 returned error can't find the container with id b724c156c701f6e470bd342c365a9276a0a2f122f1b3ea10e98b2dce91f2f017 Oct 31 19:58:11.396018 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:11.395984 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm"] Oct 31 19:58:11.396304 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:58:11.396270 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b991a69_1862_41cf_b122_6bb782044784.slice/crio-17ffd6d0f61cf124a4d13f463882122ebe60337712c840bfd00b030c91b78a19 WatchSource:0}: Error finding container 17ffd6d0f61cf124a4d13f463882122ebe60337712c840bfd00b030c91b78a19: Status 404 returned error can't find the container with id 17ffd6d0f61cf124a4d13f463882122ebe60337712c840bfd00b030c91b78a19 Oct 31 19:58:11.595679 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:11.595586 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" event={"ID":"4b991a69-1862-41cf-b122-6bb782044784","Type":"ContainerStarted","Data":"8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242"} Oct 31 19:58:11.595679 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:11.595637 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" event={"ID":"4b991a69-1862-41cf-b122-6bb782044784","Type":"ContainerStarted","Data":"17ffd6d0f61cf124a4d13f463882122ebe60337712c840bfd00b030c91b78a19"} Oct 31 19:58:11.596186 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:11.595679 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" Oct 31 19:58:11.597382 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:11.597360 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" event={"ID":"f0c12fb7-e4fd-40f1-8a3d-6909084f0394","Type":"ContainerStarted","Data":"9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe"} Oct 31 19:58:11.597489 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:11.597394 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" event={"ID":"f0c12fb7-e4fd-40f1-8a3d-6909084f0394","Type":"ContainerStarted","Data":"b724c156c701f6e470bd342c365a9276a0a2f122f1b3ea10e98b2dce91f2f017"} Oct 31 19:58:11.597489 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:11.597415 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" Oct 31 19:58:11.704563 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:11.704512 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" podStartSLOduration=1.704497961 podStartE2EDuration="1.704497961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:11.700574508 +0000 UTC m=+2463.333968413" watchObservedRunningTime="2026-10-31 19:58:11.704497961 +0000 UTC m=+2463.337891869" Oct 31 19:58:11.808717 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:11.808664 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" podStartSLOduration=1.808648847 podStartE2EDuration="1.808648847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:11.805329592 +0000 UTC m=+2463.438723502" watchObservedRunningTime="2026-10-31 19:58:11.808648847 +0000 UTC m=+2463.442042785" Oct 31 19:58:20.407313 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:58:20.407271 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:58:20.735306 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.735266 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4"] Oct 31 19:58:20.752995 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.752957 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:20.800079 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.800046 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4"] Oct 31 19:58:20.835382 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.835350 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-13e9b4-openshift-default-dockercfg-vprzs\"" Oct 31 19:58:20.955906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.952796 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-certs\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:20.955906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.952892 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-socket\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:20.955906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.952920 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwxlb\" (UniqueName: \"kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-kube-api-access-hwxlb\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:20.955906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.952984 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-podinfo\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:20.955906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.953074 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-envoy\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:20.955906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.953133 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2824e53-a817-4149-b6e4-49453b7c3d5b-istiod-ca-cert\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:20.955906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.953200 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-token\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:20.955906 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.953321 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-data\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:20.968911 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:20.961130 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-credential-socket\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.081983 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.081809 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2824e53-a817-4149-b6e4-49453b7c3d5b-istiod-ca-cert\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.081983 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.081911 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-token\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.082378 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.082012 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-data\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.082378 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.082059 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-credential-socket\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.082378 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.082131 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-certs\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.082378 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.082186 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-socket\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.082378 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.082212 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxlb\" (UniqueName: \"kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-kube-api-access-hwxlb\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.082378 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.082266 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-podinfo\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.082378 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.082380 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-envoy\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.091949 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.091397 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-credential-socket\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.092404 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.092376 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-certs\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.093148 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.093122 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-data\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.094530 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.094494 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2824e53-a817-4149-b6e4-49453b7c3d5b-istiod-ca-cert\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.094641 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.094543 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-socket\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.095873 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.095836 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-podinfo\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.108904 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.106199 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-envoy\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.148002 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.147201 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-token\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.148522 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.148500 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxlb\" (UniqueName: \"kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-kube-api-access-hwxlb\") pod \"e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.370303 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.370216 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:21.794005 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.793974 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4"] Oct 31 19:58:21.794454 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:58:21.794323 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2824e53_a817_4149_b6e4_49453b7c3d5b.slice/crio-ccd2341ced0701b965e2cca435ce225ca7b76fc572f2c80071c7770e8d75bc4c WatchSource:0}: Error finding container ccd2341ced0701b965e2cca435ce225ca7b76fc572f2c80071c7770e8d75bc4c: Status 404 returned error can't find the container with id ccd2341ced0701b965e2cca435ce225ca7b76fc572f2c80071c7770e8d75bc4c Oct 31 19:58:21.797077 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.797048 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:58:21.797150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.797114 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:58:21.797150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:21.797141 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:58:22.607162 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:22.607129 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" Oct 31 19:58:22.607532 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:22.607514 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" Oct 31 19:58:22.775633 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:22.775596 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" event={"ID":"d2824e53-a817-4149-b6e4-49453b7c3d5b","Type":"ContainerStarted","Data":"ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273"} Oct 31 19:58:22.775834 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:22.775667 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" event={"ID":"d2824e53-a817-4149-b6e4-49453b7c3d5b","Type":"ContainerStarted","Data":"ccd2341ced0701b965e2cca435ce225ca7b76fc572f2c80071c7770e8d75bc4c"} Oct 31 19:58:22.945309 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:22.945186 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" podStartSLOduration=2.9451625200000002 podStartE2EDuration="2.94516252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:22.929948706 +0000 UTC m=+2474.563342615" watchObservedRunningTime="2026-10-31 19:58:22.94516252 +0000 UTC m=+2474.578556428" Oct 31 19:58:22.981033 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:22.981004 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d"] Oct 31 19:58:22.981234 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:22.981211 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerName="istio-proxy" containerID="cri-o://fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba" gracePeriod=30 Oct 31 19:58:22.984860 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:22.984833 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h"] Oct 31 19:58:22.985104 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:22.985079 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" podUID="9fe6fc2e-343d-493f-9f8f-d3db1343e86b" containerName="istio-proxy" containerID="cri-o://13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436" gracePeriod=30 Oct 31 19:58:22.988988 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:22.988960 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4"] Oct 31 19:58:22.989193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:22.989169 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" podUID="cba362a7-7884-4660-b218-98c97194513a" containerName="istio-proxy" containerID="cri-o://2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85" gracePeriod=30 Oct 31 19:58:23.381529 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:23.370770 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:23.395541 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:23.395291 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:23.782953 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:23.782922 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:23.783784 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:23.783762 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 19:58:28.295897 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.295854 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:58:28.324150 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.324122 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:58:28.336934 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.336873 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:58:28.384707 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.384676 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-credential-socket\") pod \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " Oct 31 19:58:28.384707 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.384710 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istiod-ca-cert\") pod \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " Oct 31 19:58:28.384962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.384729 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-podinfo\") pod \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " Oct 31 19:58:28.384962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.384763 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-socket\") pod \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " Oct 31 19:58:28.384962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.384790 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-certs\") pod \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " Oct 31 19:58:28.384962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.384821 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-socket\") pod \"cba362a7-7884-4660-b218-98c97194513a\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " Oct 31 19:58:28.384962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.384850 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-certs\") pod \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " Oct 31 19:58:28.385226 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.385070 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-credential-socket" pod "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" (UID: "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.385226 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.385172 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istiod-ca-cert" pod "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" (UID: "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:58:28.385410 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.385363 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-socket" pod "cba362a7-7884-4660-b218-98c97194513a" (UID: "cba362a7-7884-4660-b218-98c97194513a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.385410 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.385381 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-socket" pod "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" (UID: "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.385410 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.385400 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-certs" pod "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" (UID: "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.385842 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.385816 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-certs" pod "9fe6fc2e-343d-493f-9f8f-d3db1343e86b" (UID: "9fe6fc2e-343d-493f-9f8f-d3db1343e86b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.385955 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.385923 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-podinfo\") pod \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " Oct 31 19:58:28.386010 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.385993 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-credential-socket\") pod \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " Oct 31 19:58:28.386070 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386050 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-kube-api-access-xlptc\" (UniqueName: \"kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-kube-api-access-xlptc\") pod \"cba362a7-7884-4660-b218-98c97194513a\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " Oct 31 19:58:28.386126 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386094 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-credential-socket\") pod \"cba362a7-7884-4660-b218-98c97194513a\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " Oct 31 19:58:28.386126 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386120 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-envoy\") pod \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " Oct 31 19:58:28.386233 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386192 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-kube-api-access-qmpzp\" (UniqueName: \"kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-kube-api-access-qmpzp\") pod \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " Oct 31 19:58:28.386233 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386203 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-credential-socket" pod "9fe6fc2e-343d-493f-9f8f-d3db1343e86b" (UID: "9fe6fc2e-343d-493f-9f8f-d3db1343e86b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.386346 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386247 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-data\") pod \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " Oct 31 19:58:28.386346 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386309 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-token\" (UniqueName: \"kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-token\") pod \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " Oct 31 19:58:28.386456 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386356 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-data\") pod \"cba362a7-7884-4660-b218-98c97194513a\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " Oct 31 19:58:28.386456 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386392 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-credential-socket" pod "cba362a7-7884-4660-b218-98c97194513a" (UID: "cba362a7-7884-4660-b218-98c97194513a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.386456 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386402 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-istio-token\" (UniqueName: \"kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-istio-token\") pod \"cba362a7-7884-4660-b218-98c97194513a\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " Oct 31 19:58:28.386456 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386449 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cba362a7-7884-4660-b218-98c97194513a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba362a7-7884-4660-b218-98c97194513a-istio-podinfo\") pod \"cba362a7-7884-4660-b218-98c97194513a\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " Oct 31 19:58:28.386709 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386514 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-token\" (UniqueName: \"kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-token\") pod \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " Oct 31 19:58:28.386709 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386543 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-data\") pod \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " Oct 31 19:58:28.386709 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386627 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-certs\") pod \"cba362a7-7884-4660-b218-98c97194513a\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " Oct 31 19:58:28.386709 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386660 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-envoy\") pod \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " Oct 31 19:58:28.386709 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386683 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-data" pod "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" (UID: "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.386709 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386705 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba362a7-7884-4660-b218-98c97194513a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba362a7-7884-4660-b218-98c97194513a-istiod-ca-cert\") pod \"cba362a7-7884-4660-b218-98c97194513a\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " Oct 31 19:58:28.387058 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386761 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-socket\") pod \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " Oct 31 19:58:28.387058 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386820 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-kube-api-access-d8zpx\" (UniqueName: \"kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-kube-api-access-d8zpx\") pod \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\" (UID: \"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c\") " Oct 31 19:58:28.387058 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386913 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-envoy\") pod \"cba362a7-7884-4660-b218-98c97194513a\" (UID: \"cba362a7-7884-4660-b218-98c97194513a\") " Oct 31 19:58:28.387058 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.386975 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istiod-ca-cert\") pod \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\" (UID: \"9fe6fc2e-343d-493f-9f8f-d3db1343e86b\") " Oct 31 19:58:28.387768 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.387524 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-podinfo" pod "9fe6fc2e-343d-493f-9f8f-d3db1343e86b" (UID: "9fe6fc2e-343d-493f-9f8f-d3db1343e86b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:58:28.387768 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.387762 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.387971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.387784 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.387971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.387797 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.387971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.387810 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.387971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.387822 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.387971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.387836 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.387971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.387849 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.387971 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.387863 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.389663 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.389620 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-podinfo" pod "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" (UID: "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:58:28.389663 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.389646 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-envoy" pod "9fe6fc2e-343d-493f-9f8f-d3db1343e86b" (UID: "9fe6fc2e-343d-493f-9f8f-d3db1343e86b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.389920 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.389859 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-data" pod "cba362a7-7884-4660-b218-98c97194513a" (UID: "cba362a7-7884-4660-b218-98c97194513a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.390295 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.390271 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba362a7-7884-4660-b218-98c97194513a-istiod-ca-cert" pod "cba362a7-7884-4660-b218-98c97194513a" (UID: "cba362a7-7884-4660-b218-98c97194513a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:58:28.390554 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.390527 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-data" pod "9fe6fc2e-343d-493f-9f8f-d3db1343e86b" (UID: "9fe6fc2e-343d-493f-9f8f-d3db1343e86b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.390705 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.390652 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-token" pod "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" (UID: "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:58:28.390825 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.390734 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-certs" pod "cba362a7-7884-4660-b218-98c97194513a" (UID: "cba362a7-7884-4660-b218-98c97194513a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.391787 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.391506 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-token" pod "9fe6fc2e-343d-493f-9f8f-d3db1343e86b" (UID: "9fe6fc2e-343d-493f-9f8f-d3db1343e86b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:58:28.391787 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.387875 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.391787 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.391724 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.391787 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.391752 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-istio-token" pod "cba362a7-7884-4660-b218-98c97194513a" (UID: "cba362a7-7884-4660-b218-98c97194513a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:58:28.391787 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.391781 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-kube-api-access-xlptc" pod "cba362a7-7884-4660-b218-98c97194513a" (UID: "cba362a7-7884-4660-b218-98c97194513a"). InnerVolumeSpecName "kube-api-access-xlptc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:58:28.392232 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.392013 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-socket" pod "9fe6fc2e-343d-493f-9f8f-d3db1343e86b" (UID: "9fe6fc2e-343d-493f-9f8f-d3db1343e86b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.392232 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.392084 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istiod-ca-cert" pod "9fe6fc2e-343d-493f-9f8f-d3db1343e86b" (UID: "9fe6fc2e-343d-493f-9f8f-d3db1343e86b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:58:28.392232 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.392155 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-kube-api-access-qmpzp" pod "9fe6fc2e-343d-493f-9f8f-d3db1343e86b" (UID: "9fe6fc2e-343d-493f-9f8f-d3db1343e86b"). InnerVolumeSpecName "kube-api-access-qmpzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:58:28.392432 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.392410 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cba362a7-7884-4660-b218-98c97194513a-istio-podinfo" pod "cba362a7-7884-4660-b218-98c97194513a" (UID: "cba362a7-7884-4660-b218-98c97194513a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:58:28.392739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.392723 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-kube-api-access-d8zpx" pod "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" (UID: "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c"). InnerVolumeSpecName "kube-api-access-d8zpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:58:28.393204 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.393181 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-envoy" pod "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" (UID: "709870d0-a4b9-4e4c-8c4b-52058d9e5e1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.394025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.394007 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-envoy" pod "cba362a7-7884-4660-b218-98c97194513a" (UID: "cba362a7-7884-4660-b218-98c97194513a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:28.492982 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.492943 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.492982 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.492976 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmpzp\" (UniqueName: \"kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-kube-api-access-qmpzp\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.492982 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.492991 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493005 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493018 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493030 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba362a7-7884-4660-b218-98c97194513a-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493043 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493056 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493069 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493082 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493095 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba362a7-7884-4660-b218-98c97194513a-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493110 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493123 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8zpx\" (UniqueName: \"kubernetes.io/projected/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-kube-api-access-d8zpx\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493135 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba362a7-7884-4660-b218-98c97194513a-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493148 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe6fc2e-343d-493f-9f8f-d3db1343e86b-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493160 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.493276 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.493174 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlptc\" (UniqueName: \"kubernetes.io/projected/cba362a7-7884-4660-b218-98c97194513a-kube-api-access-xlptc\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:28.821062 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.820973 6107 generic.go:356] "Generic (PLEG): container finished" podID="cba362a7-7884-4660-b218-98c97194513a" containerID="2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85" exitCode=0 Oct 31 19:58:28.821062 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.821023 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" Oct 31 19:58:28.821311 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.821064 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" event={"ID":"cba362a7-7884-4660-b218-98c97194513a","Type":"ContainerDied","Data":"2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85"} Oct 31 19:58:28.821311 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.821120 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4" event={"ID":"cba362a7-7884-4660-b218-98c97194513a","Type":"ContainerDied","Data":"def36d9b87a27fc4a1d3ab908ded85eebe29a336fb75370e6d27d4472efedee9"} Oct 31 19:58:28.821311 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.821142 6107 scope.go:122] "RemoveContainer" containerID="2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85" Oct 31 19:58:28.822619 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.822599 6107 generic.go:356] "Generic (PLEG): container finished" podID="9fe6fc2e-343d-493f-9f8f-d3db1343e86b" containerID="13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436" exitCode=0 Oct 31 19:58:28.822739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.822653 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" Oct 31 19:58:28.822739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.822682 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" event={"ID":"9fe6fc2e-343d-493f-9f8f-d3db1343e86b","Type":"ContainerDied","Data":"13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436"} Oct 31 19:58:28.822739 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.822735 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h" event={"ID":"9fe6fc2e-343d-493f-9f8f-d3db1343e86b","Type":"ContainerDied","Data":"4500efc6f17baf653393774285528e4ef2645087e8a2d5c14a853819215db67e"} Oct 31 19:58:28.824442 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.824421 6107 generic.go:356] "Generic (PLEG): container finished" podID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerID="fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba" exitCode=0 Oct 31 19:58:28.824564 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.824463 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" Oct 31 19:58:28.824564 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.824500 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" event={"ID":"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c","Type":"ContainerDied","Data":"fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba"} Oct 31 19:58:28.824564 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.824560 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d" event={"ID":"709870d0-a4b9-4e4c-8c4b-52058d9e5e1c","Type":"ContainerDied","Data":"2d04a66025985c533642e3b3e9f84965b2c56198511623769b2d3a948f1a096d"} Oct 31 19:58:28.833506 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.833490 6107 scope.go:122] "RemoveContainer" containerID="2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85" Oct 31 19:58:28.833805 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:58:28.833787 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85\": container with ID starting with 2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85 not found: ID does not exist" containerID="2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85" Oct 31 19:58:28.833870 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.833815 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85"} err="failed to get container status \"2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85\": rpc error: code = NotFound desc = could not find container \"2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85\": container with ID starting with 2e6226c2cf45a9796490cf9f325b3a302c3eb86fef5cb8cbefc1b473fdfc6b85 not found: ID does not exist" Oct 31 19:58:28.833870 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.833833 6107 scope.go:122] "RemoveContainer" containerID="13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436" Oct 31 19:58:28.843096 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.843079 6107 scope.go:122] "RemoveContainer" containerID="13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436" Oct 31 19:58:28.843332 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:58:28.843315 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436\": container with ID starting with 13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436 not found: ID does not exist" containerID="13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436" Oct 31 19:58:28.843382 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.843339 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436"} err="failed to get container status \"13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436\": rpc error: code = NotFound desc = could not find container \"13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436\": container with ID starting with 13bce97740059160d4ad1e48d6beebee8a659695033f8f41eade0f497ab78436 not found: ID does not exist" Oct 31 19:58:28.843382 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.843354 6107 scope.go:122] "RemoveContainer" containerID="fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba" Oct 31 19:58:28.856591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.855994 6107 scope.go:122] "RemoveContainer" containerID="fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba" Oct 31 19:58:28.856591 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:58:28.856264 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba\": container with ID starting with fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba not found: ID does not exist" containerID="fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba" Oct 31 19:58:28.856591 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.856291 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba"} err="failed to get container status \"fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba\": rpc error: code = NotFound desc = could not find container \"fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba\": container with ID starting with fce7278a372bcaa2ce6f62189dd8db3a0391707de6e9cc2aed1705370d854fba not found: ID does not exist" Oct 31 19:58:28.975101 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.975070 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d"] Oct 31 19:58:28.978689 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:28.978666 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-frv2d"] Oct 31 19:58:29.078974 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:29.078898 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h"] Oct 31 19:58:29.084002 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:29.083977 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tr92h"] Oct 31 19:58:29.185579 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:29.185544 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4"] Oct 31 19:58:29.189462 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:29.189436 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-blhc4"] Oct 31 19:58:30.347563 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.347522 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck"] Oct 31 19:58:30.348689 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.348663 6107 cpu_manager.go:414] "containerMap: removing container" podUID="cba362a7-7884-4660-b218-98c97194513a" containerName="istio-proxy" Oct 31 19:58:30.348689 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.348687 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba362a7-7884-4660-b218-98c97194513a" containerName="istio-proxy" Oct 31 19:58:30.348947 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.348741 6107 cpu_manager.go:414] "containerMap: removing container" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerName="istio-proxy" Oct 31 19:58:30.348947 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.348750 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerName="istio-proxy" Oct 31 19:58:30.348947 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.348769 6107 cpu_manager.go:414] "containerMap: removing container" podUID="9fe6fc2e-343d-493f-9f8f-d3db1343e86b" containerName="istio-proxy" Oct 31 19:58:30.348947 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.348777 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe6fc2e-343d-493f-9f8f-d3db1343e86b" containerName="istio-proxy" Oct 31 19:58:30.349145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.348973 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" containerName="istio-proxy" Oct 31 19:58:30.349145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.348990 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe6fc2e-343d-493f-9f8f-d3db1343e86b" containerName="istio-proxy" Oct 31 19:58:30.349145 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.349008 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba362a7-7884-4660-b218-98c97194513a" containerName="istio-proxy" Oct 31 19:58:30.357954 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.355634 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" Oct 31 19:58:30.403151 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.403117 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck"] Oct 31 19:58:30.448913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.443288 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-13e9b4-dockercfg-778pm\"" Oct 31 19:58:30.448913 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.443380 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-13e9b4\"" Oct 31 19:58:30.463902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.449574 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdjw\" (UniqueName: \"kubernetes.io/projected/a9933276-a77f-4e32-8a01-68df35b1813c-kube-api-access-ctdjw\") pod \"payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck\" (UID: \"a9933276-a77f-4e32-8a01-68df35b1813c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" Oct 31 19:58:30.463902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.449814 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9933276-a77f-4e32-8a01-68df35b1813c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck\" (UID: \"a9933276-a77f-4e32-8a01-68df35b1813c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" Oct 31 19:58:30.531251 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.531211 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709870d0-a4b9-4e4c-8c4b-52058d9e5e1c" path="/var/lib/kubelet/pods/709870d0-a4b9-4e4c-8c4b-52058d9e5e1c/volumes" Oct 31 19:58:30.532010 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.531988 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe6fc2e-343d-493f-9f8f-d3db1343e86b" path="/var/lib/kubelet/pods/9fe6fc2e-343d-493f-9f8f-d3db1343e86b/volumes" Oct 31 19:58:30.532684 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.532664 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba362a7-7884-4660-b218-98c97194513a" path="/var/lib/kubelet/pods/cba362a7-7884-4660-b218-98c97194513a/volumes" Oct 31 19:58:30.533619 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.533586 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c"] Oct 31 19:58:30.542429 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.542392 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" Oct 31 19:58:30.552909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.551046 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdjw\" (UniqueName: \"kubernetes.io/projected/a9933276-a77f-4e32-8a01-68df35b1813c-kube-api-access-ctdjw\") pod \"payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck\" (UID: \"a9933276-a77f-4e32-8a01-68df35b1813c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" Oct 31 19:58:30.552909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.551233 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9933276-a77f-4e32-8a01-68df35b1813c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck\" (UID: \"a9933276-a77f-4e32-8a01-68df35b1813c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" Oct 31 19:58:30.552909 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.552196 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9933276-a77f-4e32-8a01-68df35b1813c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck\" (UID: \"a9933276-a77f-4e32-8a01-68df35b1813c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" Oct 31 19:58:30.564532 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.564503 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c"] Oct 31 19:58:30.606098 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.605992 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdjw\" (UniqueName: \"kubernetes.io/projected/a9933276-a77f-4e32-8a01-68df35b1813c-kube-api-access-ctdjw\") pod \"payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck\" (UID: \"a9933276-a77f-4e32-8a01-68df35b1813c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" Oct 31 19:58:30.653081 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.653040 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj922\" (UniqueName: \"kubernetes.io/projected/54065907-30f3-4605-9bf1-e7e9174af9a7-kube-api-access-sj922\") pod \"payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c\" (UID: \"54065907-30f3-4605-9bf1-e7e9174af9a7\") " pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" Oct 31 19:58:30.653330 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.653189 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54065907-30f3-4605-9bf1-e7e9174af9a7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c\" (UID: \"54065907-30f3-4605-9bf1-e7e9174af9a7\") " pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" Oct 31 19:58:30.674354 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.674325 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" Oct 31 19:58:30.755065 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.754927 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj922\" (UniqueName: \"kubernetes.io/projected/54065907-30f3-4605-9bf1-e7e9174af9a7-kube-api-access-sj922\") pod \"payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c\" (UID: \"54065907-30f3-4605-9bf1-e7e9174af9a7\") " pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" Oct 31 19:58:30.755227 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.755067 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54065907-30f3-4605-9bf1-e7e9174af9a7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c\" (UID: \"54065907-30f3-4605-9bf1-e7e9174af9a7\") " pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" Oct 31 19:58:30.756487 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.755958 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54065907-30f3-4605-9bf1-e7e9174af9a7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c\" (UID: \"54065907-30f3-4605-9bf1-e7e9174af9a7\") " pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" Oct 31 19:58:30.808471 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.808412 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj922\" (UniqueName: \"kubernetes.io/projected/54065907-30f3-4605-9bf1-e7e9174af9a7-kube-api-access-sj922\") pod \"payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c\" (UID: \"54065907-30f3-4605-9bf1-e7e9174af9a7\") " pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" Oct 31 19:58:30.861698 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:30.861601 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" Oct 31 19:58:31.030302 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.030270 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp"] Oct 31 19:58:31.036058 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.036026 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.051672 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:58:31.051645 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9933276_a77f_4e32_8a01_68df35b1813c.slice/crio-619d062de535088c3f033de68a90793f6351f58ea8d9cf3b0af091279e6dc739 WatchSource:0}: Error finding container 619d062de535088c3f033de68a90793f6351f58ea8d9cf3b0af091279e6dc739: Status 404 returned error can't find the container with id 619d062de535088c3f033de68a90793f6351f58ea8d9cf3b0af091279e6dc739 Oct 31 19:58:31.052207 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.052183 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck"] Oct 31 19:58:31.084854 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.084827 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9bbcf7-serving-cert\"" Oct 31 19:58:31.087847 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.087821 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp"] Oct 31 19:58:31.160510 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.160484 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049cf604-6e9d-435d-8f6f-e141c1fb289b-openshift-service-ca\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.160659 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.160640 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdlft\" (UniqueName: \"kubernetes.io/projected/049cf604-6e9d-435d-8f6f-e141c1fb289b-kube-api-access-cdlft\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.160790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.160773 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-metrics-certs\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.160854 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.160838 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-maas-api-tls\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.245451 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.245415 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c"] Oct 31 19:58:31.252332 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:58:31.252303 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54065907_30f3_4605_9bf1_e7e9174af9a7.slice/crio-39b8bec3cbe255976488a9a6ebaf0a5f7a62aea51321ce28b8d32d88335a7717 WatchSource:0}: Error finding container 39b8bec3cbe255976488a9a6ebaf0a5f7a62aea51321ce28b8d32d88335a7717: Status 404 returned error can't find the container with id 39b8bec3cbe255976488a9a6ebaf0a5f7a62aea51321ce28b8d32d88335a7717 Oct 31 19:58:31.262219 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.262189 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdlft\" (UniqueName: \"kubernetes.io/projected/049cf604-6e9d-435d-8f6f-e141c1fb289b-kube-api-access-cdlft\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.262333 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.262318 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-metrics-certs\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.262393 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.262359 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-maas-api-tls\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.262479 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.262465 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049cf604-6e9d-435d-8f6f-e141c1fb289b-openshift-service-ca\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.262545 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:58:31.262525 6107 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-serving-cert: secret "maas-api-e2e-api-a-9bbcf7-serving-cert" not found Oct 31 19:58:31.262597 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:58:31.262586 6107 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-maas-api-tls podName:049cf604-6e9d-435d-8f6f-e141c1fb289b nodeName:}" failed. No retries permitted until 2026-10-31 19:58:31.762571559 +0000 UTC m=+2483.395965459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-maas-api-tls") pod "maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" (UID: "049cf604-6e9d-435d-8f6f-e141c1fb289b") : secret "maas-api-e2e-api-a-9bbcf7-serving-cert" not found Oct 31 19:58:31.263311 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.263282 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049cf604-6e9d-435d-8f6f-e141c1fb289b-openshift-service-ca\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.264796 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.264773 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-metrics-certs\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.314616 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.314580 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdlft\" (UniqueName: \"kubernetes.io/projected/049cf604-6e9d-435d-8f6f-e141c1fb289b-kube-api-access-cdlft\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.769545 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.769506 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-maas-api-tls\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.772286 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.772260 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-maas-api-tls\") pod \"maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.856588 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.856547 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" event={"ID":"a9933276-a77f-4e32-8a01-68df35b1813c","Type":"ContainerStarted","Data":"a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830"} Oct 31 19:58:31.856788 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.856618 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" event={"ID":"a9933276-a77f-4e32-8a01-68df35b1813c","Type":"ContainerStarted","Data":"619d062de535088c3f033de68a90793f6351f58ea8d9cf3b0af091279e6dc739"} Oct 31 19:58:31.856958 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.856661 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" Oct 31 19:58:31.858514 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.858490 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" event={"ID":"54065907-30f3-4605-9bf1-e7e9174af9a7","Type":"ContainerStarted","Data":"14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051"} Oct 31 19:58:31.858619 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.858536 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" event={"ID":"54065907-30f3-4605-9bf1-e7e9174af9a7","Type":"ContainerStarted","Data":"39b8bec3cbe255976488a9a6ebaf0a5f7a62aea51321ce28b8d32d88335a7717"} Oct 31 19:58:31.858619 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.858565 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" Oct 31 19:58:31.952548 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.952516 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:31.969943 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:31.969895 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" podStartSLOduration=1.969866785 podStartE2EDuration="1.969866785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:31.96269793 +0000 UTC m=+2483.596091863" watchObservedRunningTime="2026-10-31 19:58:31.969866785 +0000 UTC m=+2483.603260693" Oct 31 19:58:32.073712 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:32.073649 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" podStartSLOduration=2.073611317 podStartE2EDuration="2.073611317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:32.068128986 +0000 UTC m=+2483.701522896" watchObservedRunningTime="2026-10-31 19:58:32.073611317 +0000 UTC m=+2483.707005226" Oct 31 19:58:32.381957 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:32.381872 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp"] Oct 31 19:58:32.383690 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:58:32.383651 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049cf604_6e9d_435d_8f6f_e141c1fb289b.slice/crio-952bca6d11b5e76b5c6d96a907c4cc2f621a2f17ca4eb04ba54c1485913071ab WatchSource:0}: Error finding container 952bca6d11b5e76b5c6d96a907c4cc2f621a2f17ca4eb04ba54c1485913071ab: Status 404 returned error can't find the container with id 952bca6d11b5e76b5c6d96a907c4cc2f621a2f17ca4eb04ba54c1485913071ab Oct 31 19:58:32.385834 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:32.385812 6107 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:58:32.868940 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:32.868904 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" event={"ID":"049cf604-6e9d-435d-8f6f-e141c1fb289b","Type":"ContainerStarted","Data":"952bca6d11b5e76b5c6d96a907c4cc2f621a2f17ca4eb04ba54c1485913071ab"} Oct 31 19:58:33.879849 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:33.879813 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" event={"ID":"049cf604-6e9d-435d-8f6f-e141c1fb289b","Type":"ContainerStarted","Data":"d5e42914a208fc8a341de41bbcedfcc9a76a1b596628e4e36ab0e64ac812c90d"} Oct 31 19:58:33.880284 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:33.879925 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:33.993126 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:33.993053 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" podStartSLOduration=3.089545218 podStartE2EDuration="3.993032232s" totalImagesPullingTime="903.487014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:33.986517785 +0000 UTC m=+2485.619911694" watchObservedRunningTime="2026-10-31 19:58:33.993032232 +0000 UTC m=+2485.626426140" Oct 31 19:58:39.894089 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:39.894054 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 19:58:42.871937 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:42.871897 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" Oct 31 19:58:42.872312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:42.872099 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" Oct 31 19:58:45.985090 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:45.985056 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96"] Oct 31 19:58:45.990268 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:45.990250 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.041350 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.041314 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96"] Oct 31 19:58:46.082040 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.082010 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-650cf3-serving-cert\"" Oct 31 19:58:46.082636 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.082612 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj8zc\" (UniqueName: \"kubernetes.io/projected/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-kube-api-access-rj8zc\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.082705 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.082673 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.082843 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.082821 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.083025 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.083008 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.184141 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.184106 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.184315 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.184174 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj8zc\" (UniqueName: \"kubernetes.io/projected/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-kube-api-access-rj8zc\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.184315 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.184229 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.184400 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.184369 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.184400 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:58:46.184389 6107 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-serving-cert: secret "maas-api-e2e-shared-b-w1-650cf3-serving-cert" not found Oct 31 19:58:46.184483 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:58:46.184465 6107 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-maas-api-tls podName:d6654dc0-fdbd-44b9-adfd-1dde68a708f9 nodeName:}" failed. No retries permitted until 2026-10-31 19:58:46.684446139 +0000 UTC m=+2498.317840048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-maas-api-tls") pod "maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" (UID: "d6654dc0-fdbd-44b9-adfd-1dde68a708f9") : secret "maas-api-e2e-shared-b-w1-650cf3-serving-cert" not found Oct 31 19:58:46.185055 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.185030 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.186590 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.186567 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.244232 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.244077 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj8zc\" (UniqueName: \"kubernetes.io/projected/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-kube-api-access-rj8zc\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.691585 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.691505 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.694074 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.694042 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:46.907011 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:46.906976 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:47.282082 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:47.282045 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96"] Oct 31 19:58:47.283472 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:58:47.283444 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6654dc0_fdbd_44b9_adfd_1dde68a708f9.slice/crio-2a5c779aa7bca40d7578b9ec3107b8e1b590fbdb7f3e1b2134020fbd8f4c5094 WatchSource:0}: Error finding container 2a5c779aa7bca40d7578b9ec3107b8e1b590fbdb7f3e1b2134020fbd8f4c5094: Status 404 returned error can't find the container with id 2a5c779aa7bca40d7578b9ec3107b8e1b590fbdb7f3e1b2134020fbd8f4c5094 Oct 31 19:58:48.009617 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:48.009572 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" event={"ID":"d6654dc0-fdbd-44b9-adfd-1dde68a708f9","Type":"ContainerStarted","Data":"2a5c779aa7bca40d7578b9ec3107b8e1b590fbdb7f3e1b2134020fbd8f4c5094"} Oct 31 19:58:49.018386 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:49.018345 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" event={"ID":"d6654dc0-fdbd-44b9-adfd-1dde68a708f9","Type":"ContainerStarted","Data":"f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c"} Oct 31 19:58:49.018842 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:49.018446 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:58:49.131721 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:49.131634 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" podStartSLOduration=3.226186874 podStartE2EDuration="4.131614335s" totalImagesPullingTime="905.427461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:49.126022477 +0000 UTC m=+2500.759416388" watchObservedRunningTime="2026-10-31 19:58:49.131614335 +0000 UTC m=+2500.765008243" Oct 31 19:58:50.405794 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:58:50.405755 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:58:55.034991 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:58:55.034962 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 19:59:03.184969 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:03.184864 6107 generic.go:356] "Generic (PLEG): container finished" podID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerID="aeb1ffc08c610db47e0d29ac08780c55190d5f1a65d0bc52e67af92521a5c81c" exitCode=0 Oct 31 19:59:03.184969 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:03.184948 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" event={"ID":"63dee691-27c6-4a5f-882c-fc3ed46cbeb1","Type":"ContainerDied","Data":"aeb1ffc08c610db47e0d29ac08780c55190d5f1a65d0bc52e67af92521a5c81c"} Oct 31 19:59:03.185466 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:03.185383 6107 scope.go:122] "RemoveContainer" containerID="aeb1ffc08c610db47e0d29ac08780c55190d5f1a65d0bc52e67af92521a5c81c" Oct 31 19:59:03.187238 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:03.187172 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:59:03.187344 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:03.187251 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:59:03.187344 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:03.187290 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 19:59:04.195708 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:04.195673 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" event={"ID":"63dee691-27c6-4a5f-882c-fc3ed46cbeb1","Type":"ContainerStarted","Data":"5c59dfc0ec2509b4f8d009e1c9dfec5f2a6ab62b6141319f0ce982fb42fb3573"} Oct 31 19:59:04.362766 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:04.362736 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:59:04.363017 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:04.362992 6107 patch_prober.go:29] interesting pod/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 19:59:04.363079 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:04.363060 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 19:59:05.363444 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:05.363410 6107 patch_prober.go:29] interesting pod/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 19:59:05.363831 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:05.363464 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 19:59:06.362996 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:06.362962 6107 patch_prober.go:29] interesting pod/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 19:59:06.363164 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:06.363028 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 19:59:07.363447 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:07.363406 6107 patch_prober.go:29] interesting pod/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 19:59:07.363863 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:07.363465 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 19:59:07.941975 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:07.941054 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn"] Oct 31 19:59:07.941975 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:07.941302 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" podUID="53e32d79-e9ef-4b86-830c-2e1726c09b81" containerName="istio-proxy" containerID="cri-o://44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911" gracePeriod=30 Oct 31 19:59:07.977404 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:07.977368 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk"] Oct 31 19:59:07.977610 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:07.977577 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" podUID="736a0b55-5f01-414e-b297-6a1db400b1cf" containerName="istio-proxy" containerID="cri-o://9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4" gracePeriod=30 Oct 31 19:59:08.362610 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:08.362575 6107 patch_prober.go:29] interesting pod/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 19:59:08.362799 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:08.362632 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 19:59:09.070743 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:09.070709 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh"] Oct 31 19:59:09.071186 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:09.070936 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" podUID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" containerName="istio-proxy" containerID="cri-o://6114ec6cbfa5fccee38d92ff8713fc0d88015013f8c67b7ea3918032a02bfc9c" gracePeriod=30 Oct 31 19:59:09.363312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:09.363231 6107 patch_prober.go:29] interesting pod/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 19:59:09.363312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:09.363279 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 19:59:09.428926 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:09.428875 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:59:09.429116 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:09.428951 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" podUID="53e32d79-e9ef-4b86-830c-2e1726c09b81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:59:09.727922 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:09.727861 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:59:09.728190 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:09.728157 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" podUID="736a0b55-5f01-414e-b297-6a1db400b1cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:59:09.881538 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:09.881503 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw"] Oct 31 19:59:09.881797 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:09.881767 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" containerID="cri-o://79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772" gracePeriod=30 Oct 31 19:59:10.367732 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:10.367701 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:59:10.368247 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:10.368121 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:59:10.368790 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:10.368771 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 19:59:10.728615 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:10.728575 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:59:10.728802 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:10.728638 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" podUID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:59:12.985255 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:59:12.985229 6107 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e32d79_e9ef_4b86_830c_2e1726c09b81.slice/crio-44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911.scope\": RecentStats: unable to find data in memory cache]" Oct 31 19:59:13.266024 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.266001 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:59:13.289317 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.289292 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:59:13.308726 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.308695 6107 generic.go:356] "Generic (PLEG): container finished" podID="736a0b55-5f01-414e-b297-6a1db400b1cf" containerID="9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4" exitCode=0 Oct 31 19:59:13.308840 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.308753 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" Oct 31 19:59:13.308840 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.308782 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" event={"ID":"736a0b55-5f01-414e-b297-6a1db400b1cf","Type":"ContainerDied","Data":"9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4"} Oct 31 19:59:13.308949 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.308900 6107 scope.go:122] "RemoveContainer" containerID="9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4" Oct 31 19:59:13.308949 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.308874 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk" event={"ID":"736a0b55-5f01-414e-b297-6a1db400b1cf","Type":"ContainerDied","Data":"fd03057ffb856195b9b8977f1072f85a7ca03b590ed316c05ef9a6b4f4fedf9b"} Oct 31 19:59:13.310345 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.310322 6107 generic.go:356] "Generic (PLEG): container finished" podID="53e32d79-e9ef-4b86-830c-2e1726c09b81" containerID="44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911" exitCode=0 Oct 31 19:59:13.310457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.310371 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" Oct 31 19:59:13.310457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.310394 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" event={"ID":"53e32d79-e9ef-4b86-830c-2e1726c09b81","Type":"ContainerDied","Data":"44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911"} Oct 31 19:59:13.310457 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.310447 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn" event={"ID":"53e32d79-e9ef-4b86-830c-2e1726c09b81","Type":"ContainerDied","Data":"861de40e4c678f2941d2d01e11b7cc2bb68c32d91e15e44cfa0ff25ca087054e"} Oct 31 19:59:13.319351 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.319333 6107 scope.go:122] "RemoveContainer" containerID="9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4" Oct 31 19:59:13.319614 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:59:13.319597 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4\": container with ID starting with 9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4 not found: ID does not exist" containerID="9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4" Oct 31 19:59:13.319675 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.319624 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4"} err="failed to get container status \"9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4\": rpc error: code = NotFound desc = could not find container \"9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4\": container with ID starting with 9423c50f2032ffb8fcda5f3391c52cf832ac196c024d471a3396d5c3b44bb0e4 not found: ID does not exist" Oct 31 19:59:13.319675 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.319640 6107 scope.go:122] "RemoveContainer" containerID="44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911" Oct 31 19:59:13.329451 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.329434 6107 scope.go:122] "RemoveContainer" containerID="44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911" Oct 31 19:59:13.329746 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:59:13.329729 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911\": container with ID starting with 44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911 not found: ID does not exist" containerID="44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911" Oct 31 19:59:13.329794 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.329752 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911"} err="failed to get container status \"44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911\": rpc error: code = NotFound desc = could not find container \"44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911\": container with ID starting with 44bc732e60656059ae3afdf95ac18cc1f8183816accc7a9b73c40dfd9280f911 not found: ID does not exist" Oct 31 19:59:13.388101 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388074 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-certs\") pod \"53e32d79-e9ef-4b86-830c-2e1726c09b81\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " Oct 31 19:59:13.388260 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388131 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-certs\") pod \"736a0b55-5f01-414e-b297-6a1db400b1cf\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " Oct 31 19:59:13.388260 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388162 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e32d79-e9ef-4b86-830c-2e1726c09b81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e32d79-e9ef-4b86-830c-2e1726c09b81-istiod-ca-cert\") pod \"53e32d79-e9ef-4b86-830c-2e1726c09b81\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " Oct 31 19:59:13.388260 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388180 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-credential-socket\") pod \"53e32d79-e9ef-4b86-830c-2e1726c09b81\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " Oct 31 19:59:13.388260 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388241 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-podinfo\") pod \"53e32d79-e9ef-4b86-830c-2e1726c09b81\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " Oct 31 19:59:13.388438 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388278 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-credential-socket\") pod \"736a0b55-5f01-414e-b297-6a1db400b1cf\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " Oct 31 19:59:13.388438 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388326 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-envoy\") pod \"736a0b55-5f01-414e-b297-6a1db400b1cf\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " Oct 31 19:59:13.388438 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388357 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-certs" pod "53e32d79-e9ef-4b86-830c-2e1726c09b81" (UID: "53e32d79-e9ef-4b86-830c-2e1726c09b81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:13.388438 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388385 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-certs" pod "736a0b55-5f01-414e-b297-6a1db400b1cf" (UID: "736a0b55-5f01-414e-b297-6a1db400b1cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:13.388438 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388417 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-envoy\") pod \"53e32d79-e9ef-4b86-830c-2e1726c09b81\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " Oct 31 19:59:13.388673 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388490 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-socket\") pod \"736a0b55-5f01-414e-b297-6a1db400b1cf\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " Oct 31 19:59:13.388673 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388543 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-data\") pod \"53e32d79-e9ef-4b86-830c-2e1726c09b81\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " Oct 31 19:59:13.388673 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388579 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-token\" (UniqueName: \"kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-token\") pod \"736a0b55-5f01-414e-b297-6a1db400b1cf\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " Oct 31 19:59:13.388673 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388586 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-credential-socket" pod "53e32d79-e9ef-4b86-830c-2e1726c09b81" (UID: "53e32d79-e9ef-4b86-830c-2e1726c09b81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:13.388673 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388628 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-credential-socket" pod "736a0b55-5f01-414e-b297-6a1db400b1cf" (UID: "736a0b55-5f01-414e-b297-6a1db400b1cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:13.388673 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388649 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-kube-api-access-2bswh\" (UniqueName: \"kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-kube-api-access-2bswh\") pod \"53e32d79-e9ef-4b86-830c-2e1726c09b81\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " Oct 31 19:59:13.389031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388684 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e32d79-e9ef-4b86-830c-2e1726c09b81-istiod-ca-cert" pod "53e32d79-e9ef-4b86-830c-2e1726c09b81" (UID: "53e32d79-e9ef-4b86-830c-2e1726c09b81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:59:13.389031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388689 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-data\") pod \"736a0b55-5f01-414e-b297-6a1db400b1cf\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " Oct 31 19:59:13.389031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388778 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-kube-api-access-6rncx\" (UniqueName: \"kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-kube-api-access-6rncx\") pod \"736a0b55-5f01-414e-b297-6a1db400b1cf\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " Oct 31 19:59:13.389031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388775 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-socket" pod "736a0b55-5f01-414e-b297-6a1db400b1cf" (UID: "736a0b55-5f01-414e-b297-6a1db400b1cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:13.389031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388814 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736a0b55-5f01-414e-b297-6a1db400b1cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736a0b55-5f01-414e-b297-6a1db400b1cf-istiod-ca-cert\") pod \"736a0b55-5f01-414e-b297-6a1db400b1cf\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " Oct 31 19:59:13.389031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388854 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-socket\") pod \"53e32d79-e9ef-4b86-830c-2e1726c09b81\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " Oct 31 19:59:13.389031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388840 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-data" pod "53e32d79-e9ef-4b86-830c-2e1726c09b81" (UID: "53e32d79-e9ef-4b86-830c-2e1726c09b81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:13.389031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388906 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-data" pod "736a0b55-5f01-414e-b297-6a1db400b1cf" (UID: "736a0b55-5f01-414e-b297-6a1db400b1cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:13.389031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.388933 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-podinfo\") pod \"736a0b55-5f01-414e-b297-6a1db400b1cf\" (UID: \"736a0b55-5f01-414e-b297-6a1db400b1cf\") " Oct 31 19:59:13.389031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.389000 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-token\" (UniqueName: \"kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-token\") pod \"53e32d79-e9ef-4b86-830c-2e1726c09b81\" (UID: \"53e32d79-e9ef-4b86-830c-2e1726c09b81\") " Oct 31 19:59:13.389528 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.389391 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-socket" pod "53e32d79-e9ef-4b86-830c-2e1726c09b81" (UID: "53e32d79-e9ef-4b86-830c-2e1726c09b81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:13.389914 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.389871 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.390033 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.389921 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.390033 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.389948 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.390033 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.389962 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.390033 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.389976 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.390033 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.389989 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.390033 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.390002 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.390033 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.390015 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e32d79-e9ef-4b86-830c-2e1726c09b81-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.390033 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.390029 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.390605 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.390579 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736a0b55-5f01-414e-b297-6a1db400b1cf-istiod-ca-cert" pod "736a0b55-5f01-414e-b297-6a1db400b1cf" (UID: "736a0b55-5f01-414e-b297-6a1db400b1cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:59:13.391348 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.391298 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-envoy" pod "53e32d79-e9ef-4b86-830c-2e1726c09b81" (UID: "53e32d79-e9ef-4b86-830c-2e1726c09b81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:13.391442 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.391423 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-kube-api-access-2bswh" pod "53e32d79-e9ef-4b86-830c-2e1726c09b81" (UID: "53e32d79-e9ef-4b86-830c-2e1726c09b81"). InnerVolumeSpecName "kube-api-access-2bswh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:13.391569 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.391545 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-token" pod "736a0b55-5f01-414e-b297-6a1db400b1cf" (UID: "736a0b55-5f01-414e-b297-6a1db400b1cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:13.391569 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.391559 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-kube-api-access-6rncx" pod "736a0b55-5f01-414e-b297-6a1db400b1cf" (UID: "736a0b55-5f01-414e-b297-6a1db400b1cf"). InnerVolumeSpecName "kube-api-access-6rncx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:13.391869 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.391844 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-token" pod "53e32d79-e9ef-4b86-830c-2e1726c09b81" (UID: "53e32d79-e9ef-4b86-830c-2e1726c09b81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:13.392021 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.391999 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-envoy" pod "736a0b55-5f01-414e-b297-6a1db400b1cf" (UID: "736a0b55-5f01-414e-b297-6a1db400b1cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:13.392107 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.392090 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-podinfo" pod "53e32d79-e9ef-4b86-830c-2e1726c09b81" (UID: "53e32d79-e9ef-4b86-830c-2e1726c09b81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:59:13.392594 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.392580 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-podinfo" pod "736a0b55-5f01-414e-b297-6a1db400b1cf" (UID: "736a0b55-5f01-414e-b297-6a1db400b1cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:59:13.491517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.491445 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.491517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.491473 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bswh\" (UniqueName: \"kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-kube-api-access-2bswh\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.491517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.491483 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rncx\" (UniqueName: \"kubernetes.io/projected/736a0b55-5f01-414e-b297-6a1db400b1cf-kube-api-access-6rncx\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.491517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.491491 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736a0b55-5f01-414e-b297-6a1db400b1cf-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.491517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.491501 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.491517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.491509 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.491517 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.491517 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.491843 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.491525 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736a0b55-5f01-414e-b297-6a1db400b1cf-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.491843 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.491534 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e32d79-e9ef-4b86-830c-2e1726c09b81-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:13.730242 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.730205 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk"] Oct 31 19:59:13.735241 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.735210 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-vhfhk"] Oct 31 19:59:13.834366 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.834290 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn"] Oct 31 19:59:13.841073 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:13.841049 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-zcwgn"] Oct 31 19:59:14.336031 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.335951 6107 generic.go:356] "Generic (PLEG): container finished" podID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" containerID="6114ec6cbfa5fccee38d92ff8713fc0d88015013f8c67b7ea3918032a02bfc9c" exitCode=0 Oct 31 19:59:14.336360 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.336026 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" event={"ID":"5e2c9cc7-732a-4d51-8657-28f8eb60577e","Type":"ContainerDied","Data":"6114ec6cbfa5fccee38d92ff8713fc0d88015013f8c67b7ea3918032a02bfc9c"} Oct 31 19:59:14.362324 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.362302 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:59:14.489393 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.489357 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e32d79-e9ef-4b86-830c-2e1726c09b81" path="/var/lib/kubelet/pods/53e32d79-e9ef-4b86-830c-2e1726c09b81/volumes" Oct 31 19:59:14.490193 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.490163 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736a0b55-5f01-414e-b297-6a1db400b1cf" path="/var/lib/kubelet/pods/736a0b55-5f01-414e-b297-6a1db400b1cf/volumes" Oct 31 19:59:14.505017 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.504967 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-envoy\") pod \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " Oct 31 19:59:14.505184 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.505029 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-podinfo\") pod \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " Oct 31 19:59:14.505184 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.505082 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-data\") pod \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " Oct 31 19:59:14.505184 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.505149 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-token\" (UniqueName: \"kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-token\") pod \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " Oct 31 19:59:14.505184 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.505168 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-kube-api-access-tqnsf\" (UniqueName: \"kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-kube-api-access-tqnsf\") pod \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " Oct 31 19:59:14.505420 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.505233 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istiod-ca-cert\") pod \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " Oct 31 19:59:14.505420 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.505272 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-certs\") pod \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " Oct 31 19:59:14.505420 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.505316 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-socket\") pod \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " Oct 31 19:59:14.505585 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.505431 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-credential-socket\") pod \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\" (UID: \"5e2c9cc7-732a-4d51-8657-28f8eb60577e\") " Oct 31 19:59:14.505897 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.505772 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-data" pod "5e2c9cc7-732a-4d51-8657-28f8eb60577e" (UID: "5e2c9cc7-732a-4d51-8657-28f8eb60577e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:14.506014 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.505986 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-credential-socket" pod "5e2c9cc7-732a-4d51-8657-28f8eb60577e" (UID: "5e2c9cc7-732a-4d51-8657-28f8eb60577e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:14.506409 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.506147 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:14.506409 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.506176 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:14.506409 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.506234 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istiod-ca-cert" pod "5e2c9cc7-732a-4d51-8657-28f8eb60577e" (UID: "5e2c9cc7-732a-4d51-8657-28f8eb60577e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:59:14.506409 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.506404 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-certs" pod "5e2c9cc7-732a-4d51-8657-28f8eb60577e" (UID: "5e2c9cc7-732a-4d51-8657-28f8eb60577e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:14.506818 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.506794 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-socket" pod "5e2c9cc7-732a-4d51-8657-28f8eb60577e" (UID: "5e2c9cc7-732a-4d51-8657-28f8eb60577e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:14.508137 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.508112 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-envoy" pod "5e2c9cc7-732a-4d51-8657-28f8eb60577e" (UID: "5e2c9cc7-732a-4d51-8657-28f8eb60577e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:14.508236 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.508122 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-podinfo" pod "5e2c9cc7-732a-4d51-8657-28f8eb60577e" (UID: "5e2c9cc7-732a-4d51-8657-28f8eb60577e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:59:14.509208 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.509187 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-kube-api-access-tqnsf" pod "5e2c9cc7-732a-4d51-8657-28f8eb60577e" (UID: "5e2c9cc7-732a-4d51-8657-28f8eb60577e"). InnerVolumeSpecName "kube-api-access-tqnsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:14.509402 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.509387 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-token" pod "5e2c9cc7-732a-4d51-8657-28f8eb60577e" (UID: "5e2c9cc7-732a-4d51-8657-28f8eb60577e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:14.607337 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.607257 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:14.607337 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.607288 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:14.607337 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.607301 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:14.607337 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.607313 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:14.607337 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.607325 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:14.607337 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.607338 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:14.607645 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:14.607351 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqnsf\" (UniqueName: \"kubernetes.io/projected/5e2c9cc7-732a-4d51-8657-28f8eb60577e-kube-api-access-tqnsf\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:15.207635 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.207607 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:59:15.316989 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.316952 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-certs\") pod \"6b26a43c-9759-4e25-a559-adf43a3b0667\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " Oct 31 19:59:15.316989 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.316992 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b26a43c-9759-4e25-a559-adf43a3b0667-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b26a43c-9759-4e25-a559-adf43a3b0667-istiod-ca-cert\") pod \"6b26a43c-9759-4e25-a559-adf43a3b0667\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " Oct 31 19:59:15.317312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.317034 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-credential-socket\") pod \"6b26a43c-9759-4e25-a559-adf43a3b0667\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " Oct 31 19:59:15.317312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.317090 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-token\" (UniqueName: \"kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-token\") pod \"6b26a43c-9759-4e25-a559-adf43a3b0667\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " Oct 31 19:59:15.317312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.317149 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-socket\") pod \"6b26a43c-9759-4e25-a559-adf43a3b0667\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " Oct 31 19:59:15.317312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.317217 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-podinfo\") pod \"6b26a43c-9759-4e25-a559-adf43a3b0667\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " Oct 31 19:59:15.317312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.317264 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-kube-api-access-8sfk7\" (UniqueName: \"kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-kube-api-access-8sfk7\") pod \"6b26a43c-9759-4e25-a559-adf43a3b0667\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " Oct 31 19:59:15.317312 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.317293 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-data\") pod \"6b26a43c-9759-4e25-a559-adf43a3b0667\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " Oct 31 19:59:15.317650 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.317332 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-certs" pod "6b26a43c-9759-4e25-a559-adf43a3b0667" (UID: "6b26a43c-9759-4e25-a559-adf43a3b0667"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:15.317650 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.317496 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-envoy\") pod \"6b26a43c-9759-4e25-a559-adf43a3b0667\" (UID: \"6b26a43c-9759-4e25-a559-adf43a3b0667\") " Oct 31 19:59:15.317650 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.317530 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b26a43c-9759-4e25-a559-adf43a3b0667-istiod-ca-cert" pod "6b26a43c-9759-4e25-a559-adf43a3b0667" (UID: "6b26a43c-9759-4e25-a559-adf43a3b0667"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:59:15.317835 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.317814 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-data" pod "6b26a43c-9759-4e25-a559-adf43a3b0667" (UID: "6b26a43c-9759-4e25-a559-adf43a3b0667"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:15.318260 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.318236 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:15.318415 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.318265 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:15.318415 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.318282 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b26a43c-9759-4e25-a559-adf43a3b0667-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:15.318642 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.318480 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-credential-socket" pod "6b26a43c-9759-4e25-a559-adf43a3b0667" (UID: "6b26a43c-9759-4e25-a559-adf43a3b0667"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:15.319936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.318939 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-socket" pod "6b26a43c-9759-4e25-a559-adf43a3b0667" (UID: "6b26a43c-9759-4e25-a559-adf43a3b0667"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:15.320206 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.320184 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-kube-api-access-8sfk7" pod "6b26a43c-9759-4e25-a559-adf43a3b0667" (UID: "6b26a43c-9759-4e25-a559-adf43a3b0667"). InnerVolumeSpecName "kube-api-access-8sfk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:15.322486 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.322464 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-podinfo" pod "6b26a43c-9759-4e25-a559-adf43a3b0667" (UID: "6b26a43c-9759-4e25-a559-adf43a3b0667"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:59:15.324609 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.324574 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-envoy" pod "6b26a43c-9759-4e25-a559-adf43a3b0667" (UID: "6b26a43c-9759-4e25-a559-adf43a3b0667"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:15.331335 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.331311 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-token" pod "6b26a43c-9759-4e25-a559-adf43a3b0667" (UID: "6b26a43c-9759-4e25-a559-adf43a3b0667"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:15.347985 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.347958 6107 generic.go:356] "Generic (PLEG): container finished" podID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerID="79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772" exitCode=0 Oct 31 19:59:15.348368 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.348020 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" Oct 31 19:59:15.348368 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.348042 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" event={"ID":"6b26a43c-9759-4e25-a559-adf43a3b0667","Type":"ContainerDied","Data":"79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772"} Oct 31 19:59:15.348368 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.348123 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw" event={"ID":"6b26a43c-9759-4e25-a559-adf43a3b0667","Type":"ContainerDied","Data":"5a5a3a4ad7cc5c5cb1088b201a378d0a0923903fd5ca2ceab8dc2a0ee65d4db4"} Oct 31 19:59:15.348368 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.348157 6107 scope.go:122] "RemoveContainer" containerID="79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772" Oct 31 19:59:15.349902 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.349866 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" Oct 31 19:59:15.350008 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.349969 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh" event={"ID":"5e2c9cc7-732a-4d51-8657-28f8eb60577e","Type":"ContainerDied","Data":"18807e38bd686b03196fbca43c83be3fa5b02467a35ea978da3868d91a6778f8"} Oct 31 19:59:15.361496 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.361478 6107 scope.go:122] "RemoveContainer" containerID="79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772" Oct 31 19:59:15.361790 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:59:15.361772 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772\": container with ID starting with 79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772 not found: ID does not exist" containerID="79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772" Oct 31 19:59:15.361859 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.361803 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772"} err="failed to get container status \"79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772\": rpc error: code = NotFound desc = could not find container \"79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772\": container with ID starting with 79a175f0b90c25c52714cc33b748c069cb28688e02e5c0f5b34b697ce2f7d772 not found: ID does not exist" Oct 31 19:59:15.361859 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.361829 6107 scope.go:122] "RemoveContainer" containerID="6114ec6cbfa5fccee38d92ff8713fc0d88015013f8c67b7ea3918032a02bfc9c" Oct 31 19:59:15.419401 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.419313 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:15.419401 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.419343 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:15.419401 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.419353 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:15.419401 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.419361 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b26a43c-9759-4e25-a559-adf43a3b0667-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:15.419401 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.419370 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b26a43c-9759-4e25-a559-adf43a3b0667-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:15.419401 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.419381 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sfk7\" (UniqueName: \"kubernetes.io/projected/6b26a43c-9759-4e25-a559-adf43a3b0667-kube-api-access-8sfk7\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:15.480394 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.480359 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw"] Oct 31 19:59:15.491122 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.491087 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-nc5qw"] Oct 31 19:59:15.592246 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.592217 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh"] Oct 31 19:59:15.596949 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:15.596921 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-7rtqh"] Oct 31 19:59:16.522079 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:16.522046 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" path="/var/lib/kubelet/pods/5e2c9cc7-732a-4d51-8657-28f8eb60577e/volumes" Oct 31 19:59:16.522660 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:16.522645 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" path="/var/lib/kubelet/pods/6b26a43c-9759-4e25-a559-adf43a3b0667/volumes" Oct 31 19:59:19.252649 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:19.252566 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht"] Oct 31 19:59:19.358991 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:19.358955 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx"] Oct 31 19:59:20.407400 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:59:20.407356 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:59:24.253821 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:24.253773 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" podUID="eff631f0-dbf7-4aeb-a148-69c5f7d38c54" containerName="payload-processing" containerID="cri-o://b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93" gracePeriod=25 Oct 31 19:59:24.360080 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:24.360011 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" podUID="ea5ba5d4-7dfd-47da-ae42-649d38b8fa57" containerName="payload-pre-processing" containerID="cri-o://620e42af4a5fe61764edf10599b34aa47e706f7624a1d966e53819e2863326fd" gracePeriod=25 Oct 31 19:59:33.135452 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:33.135411 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" podUID="ea5ba5d4-7dfd-47da-ae42-649d38b8fa57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:59:33.135969 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:33.135416 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" podUID="eff631f0-dbf7-4aeb-a148-69c5f7d38c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:59:42.189569 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.189535 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522"] Oct 31 19:59:42.190632 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.190614 6107 cpu_manager.go:414] "containerMap: removing container" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" Oct 31 19:59:42.190685 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.190636 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" Oct 31 19:59:42.190685 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.190674 6107 cpu_manager.go:414] "containerMap: removing container" podUID="53e32d79-e9ef-4b86-830c-2e1726c09b81" containerName="istio-proxy" Oct 31 19:59:42.190685 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.190682 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e32d79-e9ef-4b86-830c-2e1726c09b81" containerName="istio-proxy" Oct 31 19:59:42.190781 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.190704 6107 cpu_manager.go:414] "containerMap: removing container" podUID="736a0b55-5f01-414e-b297-6a1db400b1cf" containerName="istio-proxy" Oct 31 19:59:42.190781 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.190712 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736a0b55-5f01-414e-b297-6a1db400b1cf" containerName="istio-proxy" Oct 31 19:59:42.190781 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.190754 6107 cpu_manager.go:414] "containerMap: removing container" podUID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" containerName="istio-proxy" Oct 31 19:59:42.190781 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.190762 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" containerName="istio-proxy" Oct 31 19:59:42.190954 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.190938 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="736a0b55-5f01-414e-b297-6a1db400b1cf" containerName="istio-proxy" Oct 31 19:59:42.190989 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.190957 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e2c9cc7-732a-4d51-8657-28f8eb60577e" containerName="istio-proxy" Oct 31 19:59:42.190989 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.190970 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b26a43c-9759-4e25-a559-adf43a3b0667" containerName="istio-proxy" Oct 31 19:59:42.190989 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.190983 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e32d79-e9ef-4b86-830c-2e1726c09b81" containerName="istio-proxy" Oct 31 19:59:42.194748 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.194730 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.218653 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.218626 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2bd\" (UniqueName: \"kubernetes.io/projected/63d0bee6-b082-4ab9-af16-d903398f9696-kube-api-access-xn2bd\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.218783 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.218665 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-maas-api-tls\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.218783 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.218691 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-metrics-certs\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.218983 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.218962 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d0bee6-b082-4ab9-af16-d903398f9696-openshift-service-ca\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.247699 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.247675 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522"] Oct 31 19:59:42.287652 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.287616 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dfe658-serving-cert\"" Oct 31 19:59:42.320574 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.320506 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d0bee6-b082-4ab9-af16-d903398f9696-openshift-service-ca\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.320735 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.320688 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2bd\" (UniqueName: \"kubernetes.io/projected/63d0bee6-b082-4ab9-af16-d903398f9696-kube-api-access-xn2bd\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.320735 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.320725 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-maas-api-tls\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.320828 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.320750 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-metrics-certs\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.320905 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:59:42.320865 6107 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-serving-cert: secret "maas-api-e2e-api-a-dfe658-serving-cert" not found Oct 31 19:59:42.320976 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:59:42.320959 6107 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-maas-api-tls podName:63d0bee6-b082-4ab9-af16-d903398f9696 nodeName:}" failed. No retries permitted until 2026-10-31 19:59:42.820937733 +0000 UTC m=+2554.454331620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-maas-api-tls") pod "maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" (UID: "63d0bee6-b082-4ab9-af16-d903398f9696") : secret "maas-api-e2e-api-a-dfe658-serving-cert" not found Oct 31 19:59:42.321266 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.321244 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d0bee6-b082-4ab9-af16-d903398f9696-openshift-service-ca\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.323876 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.323849 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-metrics-certs\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.376385 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.376357 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2bd\" (UniqueName: \"kubernetes.io/projected/63d0bee6-b082-4ab9-af16-d903398f9696-kube-api-access-xn2bd\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.828226 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.828147 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-maas-api-tls\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:42.830703 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:42.830677 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-maas-api-tls\") pod \"maas-api-e2e-api-a-dfe658-6585c8dfb9-59522\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:43.109687 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:43.109567 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:43.135924 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:43.135857 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" podUID="ea5ba5d4-7dfd-47da-ae42-649d38b8fa57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:59:43.136062 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:43.135979 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" podUID="eff631f0-dbf7-4aeb-a148-69c5f7d38c54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:59:43.480124 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:43.480084 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522"] Oct 31 19:59:43.480511 ip-10-0-128-164 kubenswrapper[6107]: W1031 19:59:43.480435 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d0bee6_b082_4ab9_af16_d903398f9696.slice/crio-f5dacde436e55507c458964cfde6c1a961cca6d5bc742cfb344d95d6e16ef58c WatchSource:0}: Error finding container f5dacde436e55507c458964cfde6c1a961cca6d5bc742cfb344d95d6e16ef58c: Status 404 returned error can't find the container with id f5dacde436e55507c458964cfde6c1a961cca6d5bc742cfb344d95d6e16ef58c Oct 31 19:59:43.598446 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:43.598409 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" event={"ID":"63d0bee6-b082-4ab9-af16-d903398f9696","Type":"ContainerStarted","Data":"f5dacde436e55507c458964cfde6c1a961cca6d5bc742cfb344d95d6e16ef58c"} Oct 31 19:59:44.569523 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.569495 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" Oct 31 19:59:44.613936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.612359 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" event={"ID":"63d0bee6-b082-4ab9-af16-d903398f9696","Type":"ContainerStarted","Data":"7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1"} Oct 31 19:59:44.613936 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.613186 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 19:59:44.621290 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.621258 6107 generic.go:356] "Generic (PLEG): container finished" podID="ea5ba5d4-7dfd-47da-ae42-649d38b8fa57" containerID="620e42af4a5fe61764edf10599b34aa47e706f7624a1d966e53819e2863326fd" exitCode=0 Oct 31 19:59:44.621580 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.621335 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" event={"ID":"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57","Type":"ContainerDied","Data":"620e42af4a5fe61764edf10599b34aa47e706f7624a1d966e53819e2863326fd"} Oct 31 19:59:44.624352 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.623596 6107 generic.go:356] "Generic (PLEG): container finished" podID="eff631f0-dbf7-4aeb-a148-69c5f7d38c54" containerID="b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93" exitCode=0 Oct 31 19:59:44.624352 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.623678 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" event={"ID":"eff631f0-dbf7-4aeb-a148-69c5f7d38c54","Type":"ContainerDied","Data":"b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93"} Oct 31 19:59:44.624352 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.623708 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" event={"ID":"eff631f0-dbf7-4aeb-a148-69c5f7d38c54","Type":"ContainerDied","Data":"0dd8739c1cb14cf19781030a4728bf07087dc713aed2ba7bc02ae80f95f17a0c"} Oct 31 19:59:44.624352 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.623727 6107 scope.go:122] "RemoveContainer" containerID="b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93" Oct 31 19:59:44.624352 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.623826 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht" Oct 31 19:59:44.651974 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.651951 6107 scope.go:122] "RemoveContainer" containerID="b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93" Oct 31 19:59:44.652337 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:59:44.652312 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93\": container with ID starting with b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93 not found: ID does not exist" containerID="b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93" Oct 31 19:59:44.652446 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.652343 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93"} err="failed to get container status \"b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93\": rpc error: code = NotFound desc = could not find container \"b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93\": container with ID starting with b2f7d0fa050292a2f2ae5d52ff351665bfcb865269f38d7e950ba0f609c95d93 not found: ID does not exist" Oct 31 19:59:44.654741 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.654716 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-plugins-config-volume\") pod \"eff631f0-dbf7-4aeb-a148-69c5f7d38c54\" (UID: \"eff631f0-dbf7-4aeb-a148-69c5f7d38c54\") " Oct 31 19:59:44.655103 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.655017 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-kube-api-access-2tw2z\" (UniqueName: \"kubernetes.io/projected/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-kube-api-access-2tw2z\") pod \"eff631f0-dbf7-4aeb-a148-69c5f7d38c54\" (UID: \"eff631f0-dbf7-4aeb-a148-69c5f7d38c54\") " Oct 31 19:59:44.656242 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.656219 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-plugins-config-volume" pod "eff631f0-dbf7-4aeb-a148-69c5f7d38c54" (UID: "eff631f0-dbf7-4aeb-a148-69c5f7d38c54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:59:44.659055 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.659028 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-kube-api-access-2tw2z" pod "eff631f0-dbf7-4aeb-a148-69c5f7d38c54" (UID: "eff631f0-dbf7-4aeb-a148-69c5f7d38c54"). InnerVolumeSpecName "kube-api-access-2tw2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:44.701511 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.701485 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" Oct 31 19:59:44.756472 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.756430 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-plugins-config-volume\") pod \"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57\" (UID: \"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57\") " Oct 31 19:59:44.756706 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.756688 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-kube-api-access-vn4dw\" (UniqueName: \"kubernetes.io/projected/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-kube-api-access-vn4dw\") pod \"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57\" (UID: \"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57\") " Oct 31 19:59:44.756810 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.756788 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-plugins-config-volume" pod "ea5ba5d4-7dfd-47da-ae42-649d38b8fa57" (UID: "ea5ba5d4-7dfd-47da-ae42-649d38b8fa57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:59:44.757337 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.757317 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tw2z\" (UniqueName: \"kubernetes.io/projected/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-kube-api-access-2tw2z\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:44.757418 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.757341 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:44.757418 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.757357 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff631f0-dbf7-4aeb-a148-69c5f7d38c54-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:44.758962 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.758934 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-kube-api-access-vn4dw" pod "ea5ba5d4-7dfd-47da-ae42-649d38b8fa57" (UID: "ea5ba5d4-7dfd-47da-ae42-649d38b8fa57"). InnerVolumeSpecName "kube-api-access-vn4dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:44.808873 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.808817 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" podStartSLOduration=1.929514659 podStartE2EDuration="2.808801682s" totalImagesPullingTime="879.287023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:44.805633685 +0000 UTC m=+2556.439027592" watchObservedRunningTime="2026-10-31 19:59:44.808801682 +0000 UTC m=+2556.442195588" Oct 31 19:59:44.858573 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:44.858535 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn4dw\" (UniqueName: \"kubernetes.io/projected/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57-kube-api-access-vn4dw\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:45.069175 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:45.069108 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht"] Oct 31 19:59:45.073111 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:45.073084 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902ffd9-56fbd8c457-6vxht"] Oct 31 19:59:45.634459 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:45.634431 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" Oct 31 19:59:45.635002 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:45.634431 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx" event={"ID":"ea5ba5d4-7dfd-47da-ae42-649d38b8fa57","Type":"ContainerDied","Data":"5e99328c26354bb1b51ceffacc79b3397a7c3fb850c5c70f1f0316e4492d8d64"} Oct 31 19:59:45.635002 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:45.634589 6107 scope.go:122] "RemoveContainer" containerID="620e42af4a5fe61764edf10599b34aa47e706f7624a1d966e53819e2863326fd" Oct 31 19:59:45.787912 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:45.787851 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx"] Oct 31 19:59:45.833444 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:45.833405 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902ffd9-6f6765bc55-zjwnx"] Oct 31 19:59:46.490532 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:46.490496 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5ba5d4-7dfd-47da-ae42-649d38b8fa57" path="/var/lib/kubelet/pods/ea5ba5d4-7dfd-47da-ae42-649d38b8fa57/volumes" Oct 31 19:59:46.490950 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:46.490936 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff631f0-dbf7-4aeb-a148-69c5f7d38c54" path="/var/lib/kubelet/pods/eff631f0-dbf7-4aeb-a148-69c5f7d38c54/volumes" Oct 31 19:59:50.404021 ip-10-0-128-164 kubenswrapper[6107]: E1031 19:59:50.403982 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 19:59:51.652339 ip-10-0-128-164 kubenswrapper[6107]: I1031 19:59:51.652310 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 20:00:00.182907 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.182835 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm"] Oct 31 20:00:00.183601 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.183581 6107 cpu_manager.go:414] "containerMap: removing container" podUID="eff631f0-dbf7-4aeb-a148-69c5f7d38c54" containerName="payload-processing" Oct 31 20:00:00.183601 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.183600 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff631f0-dbf7-4aeb-a148-69c5f7d38c54" containerName="payload-processing" Oct 31 20:00:00.183741 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.183616 6107 cpu_manager.go:414] "containerMap: removing container" podUID="ea5ba5d4-7dfd-47da-ae42-649d38b8fa57" containerName="payload-pre-processing" Oct 31 20:00:00.183741 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.183621 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5ba5d4-7dfd-47da-ae42-649d38b8fa57" containerName="payload-pre-processing" Oct 31 20:00:00.183846 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.183746 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff631f0-dbf7-4aeb-a148-69c5f7d38c54" containerName="payload-processing" Oct 31 20:00:00.183846 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.183756 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5ba5d4-7dfd-47da-ae42-649d38b8fa57" containerName="payload-pre-processing" Oct 31 20:00:00.189047 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.189025 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx"] Oct 31 20:00:00.189170 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.189143 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" Oct 31 20:00:00.194015 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.193987 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" Oct 31 20:00:00.206801 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.206782 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns"] Oct 31 20:00:00.211874 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.211856 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" Oct 31 20:00:00.228997 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.228976 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv"] Oct 31 20:00:00.233294 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.233275 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv"] Oct 31 20:00:00.233424 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.233405 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" Oct 31 20:00:00.241831 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.240790 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc"] Oct 31 20:00:00.242623 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.242599 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" Oct 31 20:00:00.246182 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.246155 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh"] Oct 31 20:00:00.246861 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.246838 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" Oct 31 20:00:00.250967 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.250947 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" Oct 31 20:00:00.271048 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.271025 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:00:00.282264 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282242 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7f5\" (UniqueName: \"kubernetes.io/projected/3c3a32c2-060e-4568-9744-1dd49c0b4cba-kube-api-access-qb7f5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx\" (UID: \"3c3a32c2-060e-4568-9744-1dd49c0b4cba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" Oct 31 20:00:00.282351 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282312 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c26d2d-3fa6-403e-9aa1-611dff153b14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm\" (UID: \"60c26d2d-3fa6-403e-9aa1-611dff153b14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" Oct 31 20:00:00.282351 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282343 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm8tc\" (UniqueName: \"kubernetes.io/projected/620ec382-3553-4b53-8e0e-355469f9b0e5-kube-api-access-wm8tc\") pod \"maas-api-key-cleanup-29891280-mhpjv\" (UID: \"620ec382-3553-4b53-8e0e-355469f9b0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" Oct 31 20:00:00.282423 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282368 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh\" (UID: \"ed0e177d-af3a-41e8-bcd8-e963493aa6ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" Oct 31 20:00:00.282505 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282475 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwz7d\" (UniqueName: \"kubernetes.io/projected/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-kube-api-access-qwz7d\") pod \"maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh\" (UID: \"ed0e177d-af3a-41e8-bcd8-e963493aa6ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" Oct 31 20:00:00.282607 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282514 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3a32c2-060e-4568-9744-1dd49c0b4cba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx\" (UID: \"3c3a32c2-060e-4568-9744-1dd49c0b4cba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" Oct 31 20:00:00.282607 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282575 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns\" (UID: \"29dfb5f8-7c03-427a-aa3c-4181855ed5fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" Oct 31 20:00:00.282607 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282604 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8vv\" (UniqueName: \"kubernetes.io/projected/d68b1ca1-2e79-4786-9799-94151203f153-kube-api-access-nt8vv\") pod \"maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv\" (UID: \"d68b1ca1-2e79-4786-9799-94151203f153\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" Oct 31 20:00:00.282710 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282633 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ed662e-cdac-4394-ac95-4b103b48bbff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc\" (UID: \"40ed662e-cdac-4394-ac95-4b103b48bbff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" Oct 31 20:00:00.282747 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282726 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b1ca1-2e79-4786-9799-94151203f153-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv\" (UID: \"d68b1ca1-2e79-4786-9799-94151203f153\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" Oct 31 20:00:00.282782 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282750 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h868g\" (UniqueName: \"kubernetes.io/projected/40ed662e-cdac-4394-ac95-4b103b48bbff-kube-api-access-h868g\") pod \"maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc\" (UID: \"40ed662e-cdac-4394-ac95-4b103b48bbff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" Oct 31 20:00:00.282782 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282777 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620ec382-3553-4b53-8e0e-355469f9b0e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891280-mhpjv\" (UID: \"620ec382-3553-4b53-8e0e-355469f9b0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" Oct 31 20:00:00.283009 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.282986 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4phg\" (UniqueName: \"kubernetes.io/projected/60c26d2d-3fa6-403e-9aa1-611dff153b14-kube-api-access-x4phg\") pod \"maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm\" (UID: \"60c26d2d-3fa6-403e-9aa1-611dff153b14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" Oct 31 20:00:00.283105 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.283020 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgnr\" (UniqueName: \"kubernetes.io/projected/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-kube-api-access-6hgnr\") pod \"maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns\" (UID: \"29dfb5f8-7c03-427a-aa3c-4181855ed5fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" Oct 31 20:00:00.303213 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.302929 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm"] Oct 31 20:00:00.317923 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.317894 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx"] Oct 31 20:00:00.359933 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.357328 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv"] Oct 31 20:00:00.359933 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.359638 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv"] Oct 31 20:00:00.362642 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.362614 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh"] Oct 31 20:00:00.367161 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.367135 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc"] Oct 31 20:00:00.384271 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384244 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4phg\" (UniqueName: \"kubernetes.io/projected/60c26d2d-3fa6-403e-9aa1-611dff153b14-kube-api-access-x4phg\") pod \"maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm\" (UID: \"60c26d2d-3fa6-403e-9aa1-611dff153b14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" Oct 31 20:00:00.384402 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384273 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgnr\" (UniqueName: \"kubernetes.io/projected/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-kube-api-access-6hgnr\") pod \"maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns\" (UID: \"29dfb5f8-7c03-427a-aa3c-4181855ed5fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" Oct 31 20:00:00.384492 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384469 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7f5\" (UniqueName: \"kubernetes.io/projected/3c3a32c2-060e-4568-9744-1dd49c0b4cba-kube-api-access-qb7f5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx\" (UID: \"3c3a32c2-060e-4568-9744-1dd49c0b4cba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" Oct 31 20:00:00.384577 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384560 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c26d2d-3fa6-403e-9aa1-611dff153b14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm\" (UID: \"60c26d2d-3fa6-403e-9aa1-611dff153b14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" Oct 31 20:00:00.384642 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384604 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm8tc\" (UniqueName: \"kubernetes.io/projected/620ec382-3553-4b53-8e0e-355469f9b0e5-kube-api-access-wm8tc\") pod \"maas-api-key-cleanup-29891280-mhpjv\" (UID: \"620ec382-3553-4b53-8e0e-355469f9b0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" Oct 31 20:00:00.384699 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384642 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh\" (UID: \"ed0e177d-af3a-41e8-bcd8-e963493aa6ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" Oct 31 20:00:00.384699 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384678 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwz7d\" (UniqueName: \"kubernetes.io/projected/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-kube-api-access-qwz7d\") pod \"maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh\" (UID: \"ed0e177d-af3a-41e8-bcd8-e963493aa6ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" Oct 31 20:00:00.384813 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384708 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3a32c2-060e-4568-9744-1dd49c0b4cba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx\" (UID: \"3c3a32c2-060e-4568-9744-1dd49c0b4cba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" Oct 31 20:00:00.384813 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384747 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns\" (UID: \"29dfb5f8-7c03-427a-aa3c-4181855ed5fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" Oct 31 20:00:00.384813 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384782 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8vv\" (UniqueName: \"kubernetes.io/projected/d68b1ca1-2e79-4786-9799-94151203f153-kube-api-access-nt8vv\") pod \"maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv\" (UID: \"d68b1ca1-2e79-4786-9799-94151203f153\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" Oct 31 20:00:00.384997 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384826 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ed662e-cdac-4394-ac95-4b103b48bbff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc\" (UID: \"40ed662e-cdac-4394-ac95-4b103b48bbff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" Oct 31 20:00:00.384997 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384935 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b1ca1-2e79-4786-9799-94151203f153-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv\" (UID: \"d68b1ca1-2e79-4786-9799-94151203f153\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" Oct 31 20:00:00.384997 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.384965 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h868g\" (UniqueName: \"kubernetes.io/projected/40ed662e-cdac-4394-ac95-4b103b48bbff-kube-api-access-h868g\") pod \"maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc\" (UID: \"40ed662e-cdac-4394-ac95-4b103b48bbff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" Oct 31 20:00:00.385157 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.385005 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620ec382-3553-4b53-8e0e-355469f9b0e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891280-mhpjv\" (UID: \"620ec382-3553-4b53-8e0e-355469f9b0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" Oct 31 20:00:00.385311 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.385288 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c26d2d-3fa6-403e-9aa1-611dff153b14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm\" (UID: \"60c26d2d-3fa6-403e-9aa1-611dff153b14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" Oct 31 20:00:00.385750 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.385727 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh\" (UID: \"ed0e177d-af3a-41e8-bcd8-e963493aa6ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" Oct 31 20:00:00.385802 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.385737 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620ec382-3553-4b53-8e0e-355469f9b0e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891280-mhpjv\" (UID: \"620ec382-3553-4b53-8e0e-355469f9b0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" Oct 31 20:00:00.386267 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.386245 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3a32c2-060e-4568-9744-1dd49c0b4cba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx\" (UID: \"3c3a32c2-060e-4568-9744-1dd49c0b4cba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" Oct 31 20:00:00.386440 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.386413 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ed662e-cdac-4394-ac95-4b103b48bbff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc\" (UID: \"40ed662e-cdac-4394-ac95-4b103b48bbff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" Oct 31 20:00:00.386533 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.386438 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns\" (UID: \"29dfb5f8-7c03-427a-aa3c-4181855ed5fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" Oct 31 20:00:00.386960 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.386938 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b1ca1-2e79-4786-9799-94151203f153-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv\" (UID: \"d68b1ca1-2e79-4786-9799-94151203f153\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" Oct 31 20:00:00.391919 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.391893 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns"] Oct 31 20:00:00.438354 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.438274 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7f5\" (UniqueName: \"kubernetes.io/projected/3c3a32c2-060e-4568-9744-1dd49c0b4cba-kube-api-access-qb7f5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx\" (UID: \"3c3a32c2-060e-4568-9744-1dd49c0b4cba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" Oct 31 20:00:00.438839 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.438808 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgnr\" (UniqueName: \"kubernetes.io/projected/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-kube-api-access-6hgnr\") pod \"maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns\" (UID: \"29dfb5f8-7c03-427a-aa3c-4181855ed5fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" Oct 31 20:00:00.439074 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.439052 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwz7d\" (UniqueName: \"kubernetes.io/projected/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-kube-api-access-qwz7d\") pod \"maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh\" (UID: \"ed0e177d-af3a-41e8-bcd8-e963493aa6ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" Oct 31 20:00:00.439744 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.439717 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4phg\" (UniqueName: \"kubernetes.io/projected/60c26d2d-3fa6-403e-9aa1-611dff153b14-kube-api-access-x4phg\") pod \"maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm\" (UID: \"60c26d2d-3fa6-403e-9aa1-611dff153b14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" Oct 31 20:00:00.439873 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.439765 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8vv\" (UniqueName: \"kubernetes.io/projected/d68b1ca1-2e79-4786-9799-94151203f153-kube-api-access-nt8vv\") pod \"maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv\" (UID: \"d68b1ca1-2e79-4786-9799-94151203f153\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" Oct 31 20:00:00.442990 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.442959 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm8tc\" (UniqueName: \"kubernetes.io/projected/620ec382-3553-4b53-8e0e-355469f9b0e5-kube-api-access-wm8tc\") pod \"maas-api-key-cleanup-29891280-mhpjv\" (UID: \"620ec382-3553-4b53-8e0e-355469f9b0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" Oct 31 20:00:00.452963 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.444042 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h868g\" (UniqueName: \"kubernetes.io/projected/40ed662e-cdac-4394-ac95-4b103b48bbff-kube-api-access-h868g\") pod \"maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc\" (UID: \"40ed662e-cdac-4394-ac95-4b103b48bbff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" Oct 31 20:00:00.507054 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.507026 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" Oct 31 20:00:00.513256 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.513231 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" Oct 31 20:00:00.529394 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.529220 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" Oct 31 20:00:00.555058 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.555033 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" Oct 31 20:00:00.562796 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.562294 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" Oct 31 20:00:00.573961 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.573837 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" Oct 31 20:00:00.578519 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.578380 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" Oct 31 20:00:00.921840 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:00.921810 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm"] Oct 31 20:00:01.129353 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:00:01.129328 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3a32c2_060e_4568_9744_1dd49c0b4cba.slice/crio-e126e38680933fab4163d58e831c45537d7e7d8bfe2acbd6fa1e3ba13643c2d7 WatchSource:0}: Error finding container e126e38680933fab4163d58e831c45537d7e7d8bfe2acbd6fa1e3ba13643c2d7: Status 404 returned error can't find the container with id e126e38680933fab4163d58e831c45537d7e7d8bfe2acbd6fa1e3ba13643c2d7 Oct 31 20:00:01.129980 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.129952 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx"] Oct 31 20:00:01.161372 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.161350 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns"] Oct 31 20:00:01.170118 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:00:01.170089 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dfb5f8_7c03_427a_aa3c_4181855ed5fa.slice/crio-928cd943075f762ef120195941edce924de7b5f3b50b8376c0447ef2cd6de0ec WatchSource:0}: Error finding container 928cd943075f762ef120195941edce924de7b5f3b50b8376c0447ef2cd6de0ec: Status 404 returned error can't find the container with id 928cd943075f762ef120195941edce924de7b5f3b50b8376c0447ef2cd6de0ec Oct 31 20:00:01.171817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.171794 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc"] Oct 31 20:00:01.171954 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:00:01.171889 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ed662e_cdac_4394_ac95_4b103b48bbff.slice/crio-5909b57b538e005046c910450f66ebe233f9a71faf3a17b823e60df6522401bf WatchSource:0}: Error finding container 5909b57b538e005046c910450f66ebe233f9a71faf3a17b823e60df6522401bf: Status 404 returned error can't find the container with id 5909b57b538e005046c910450f66ebe233f9a71faf3a17b823e60df6522401bf Oct 31 20:00:01.189300 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.189269 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv"] Oct 31 20:00:01.191009 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:00:01.190978 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68b1ca1_2e79_4786_9799_94151203f153.slice/crio-ae19e6589aeb70da7d58e9a8488d9918ccca69155218ea9c9d3489b2fe38e55f WatchSource:0}: Error finding container ae19e6589aeb70da7d58e9a8488d9918ccca69155218ea9c9d3489b2fe38e55f: Status 404 returned error can't find the container with id ae19e6589aeb70da7d58e9a8488d9918ccca69155218ea9c9d3489b2fe38e55f Oct 31 20:00:01.253211 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:00:01.253180 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0e177d_af3a_41e8_bcd8_e963493aa6ca.slice/crio-abab73c525800064f6d0de1e5cf2e03c20242c1eaad5ccd755b3bbfd0e37b466 WatchSource:0}: Error finding container abab73c525800064f6d0de1e5cf2e03c20242c1eaad5ccd755b3bbfd0e37b466: Status 404 returned error can't find the container with id abab73c525800064f6d0de1e5cf2e03c20242c1eaad5ccd755b3bbfd0e37b466 Oct 31 20:00:01.254000 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.253976 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh"] Oct 31 20:00:01.282922 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.282745 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv"] Oct 31 20:00:01.797929 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.797796 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" event={"ID":"3c3a32c2-060e-4568-9744-1dd49c0b4cba","Type":"ContainerStarted","Data":"0775c43e92b8e0940189acee9d7deed4ed40c5704ffe8b5654425ad641e961cc"} Oct 31 20:00:01.797929 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.797858 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" event={"ID":"3c3a32c2-060e-4568-9744-1dd49c0b4cba","Type":"ContainerStarted","Data":"e126e38680933fab4163d58e831c45537d7e7d8bfe2acbd6fa1e3ba13643c2d7"} Oct 31 20:00:01.799757 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.799727 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" event={"ID":"620ec382-3553-4b53-8e0e-355469f9b0e5","Type":"ContainerStarted","Data":"6abd6e334b1691350f60a51258ef71ae09c87cffe088e6defba54a9bbcb95c08"} Oct 31 20:00:01.803959 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.803928 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" event={"ID":"ed0e177d-af3a-41e8-bcd8-e963493aa6ca","Type":"ContainerStarted","Data":"f07a80922df9173b7441eb4e02486332db60cec6184bf5ee90639ca9c0cc9601"} Oct 31 20:00:01.804097 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.803983 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" event={"ID":"ed0e177d-af3a-41e8-bcd8-e963493aa6ca","Type":"ContainerStarted","Data":"abab73c525800064f6d0de1e5cf2e03c20242c1eaad5ccd755b3bbfd0e37b466"} Oct 31 20:00:01.805616 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.805583 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" event={"ID":"29dfb5f8-7c03-427a-aa3c-4181855ed5fa","Type":"ContainerStarted","Data":"675854e35e3f2b674f0d05761aa304538aab68f571812518512d1e4b3c6293fd"} Oct 31 20:00:01.805763 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.805656 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" event={"ID":"29dfb5f8-7c03-427a-aa3c-4181855ed5fa","Type":"ContainerStarted","Data":"928cd943075f762ef120195941edce924de7b5f3b50b8376c0447ef2cd6de0ec"} Oct 31 20:00:01.807415 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.807394 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" event={"ID":"60c26d2d-3fa6-403e-9aa1-611dff153b14","Type":"ContainerStarted","Data":"1ac04b2a63b29670efce4d05f8eb817a08bb499bb7cca41a540d142f361877f4"} Oct 31 20:00:01.807537 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.807439 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" event={"ID":"60c26d2d-3fa6-403e-9aa1-611dff153b14","Type":"ContainerStarted","Data":"9e46e5f5f219cd1aad102c1c438b2551b8b9d583532916897e1bd1093ce10000"} Oct 31 20:00:01.809046 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.809018 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" event={"ID":"40ed662e-cdac-4394-ac95-4b103b48bbff","Type":"ContainerStarted","Data":"5909b57b538e005046c910450f66ebe233f9a71faf3a17b823e60df6522401bf"} Oct 31 20:00:01.810915 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.810844 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" event={"ID":"d68b1ca1-2e79-4786-9799-94151203f153","Type":"ContainerStarted","Data":"7eba3b1f01f0537b07c8a348d66e2111934b30c400879aa4810f767a65cd9359"} Oct 31 20:00:01.810997 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.810945 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" event={"ID":"d68b1ca1-2e79-4786-9799-94151203f153","Type":"ContainerStarted","Data":"ae19e6589aeb70da7d58e9a8488d9918ccca69155218ea9c9d3489b2fe38e55f"} Oct 31 20:00:01.980628 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:01.980572 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" podStartSLOduration=1.9805539620000001 podStartE2EDuration="1.980553962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:01.959358992 +0000 UTC m=+2573.592752903" watchObservedRunningTime="2026-10-31 20:00:01.980553962 +0000 UTC m=+2573.613947872" Oct 31 20:00:02.068086 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.067959 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" podStartSLOduration=2.067941559 podStartE2EDuration="2.067941559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.063727839 +0000 UTC m=+2573.697121747" watchObservedRunningTime="2026-10-31 20:00:02.067941559 +0000 UTC m=+2573.701335483" Oct 31 20:00:02.171265 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.171207 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" podStartSLOduration=2.171193385 podStartE2EDuration="2.171193385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.168120484 +0000 UTC m=+2573.801514392" watchObservedRunningTime="2026-10-31 20:00:02.171193385 +0000 UTC m=+2573.804587320" Oct 31 20:00:02.276160 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.276109 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" podStartSLOduration=2.276093223 podStartE2EDuration="2.276093223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.271835999 +0000 UTC m=+2573.905229906" watchObservedRunningTime="2026-10-31 20:00:02.276093223 +0000 UTC m=+2573.909487128" Oct 31 20:00:02.378527 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.378474 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" podStartSLOduration=2.378458794 podStartE2EDuration="2.378458794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.376449711 +0000 UTC m=+2574.009843629" watchObservedRunningTime="2026-10-31 20:00:02.378458794 +0000 UTC m=+2574.011852703" Oct 31 20:00:02.822238 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.822198 6107 generic.go:356] "Generic (PLEG): container finished" podID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" containerID="675854e35e3f2b674f0d05761aa304538aab68f571812518512d1e4b3c6293fd" exitCode=60 Oct 31 20:00:02.822433 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.822288 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" event={"ID":"29dfb5f8-7c03-427a-aa3c-4181855ed5fa","Type":"ContainerDied","Data":"675854e35e3f2b674f0d05761aa304538aab68f571812518512d1e4b3c6293fd"} Oct 31 20:00:02.822670 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.822654 6107 scope.go:122] "RemoveContainer" containerID="675854e35e3f2b674f0d05761aa304538aab68f571812518512d1e4b3c6293fd" Oct 31 20:00:02.824146 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.824113 6107 generic.go:356] "Generic (PLEG): container finished" podID="60c26d2d-3fa6-403e-9aa1-611dff153b14" containerID="1ac04b2a63b29670efce4d05f8eb817a08bb499bb7cca41a540d142f361877f4" exitCode=60 Oct 31 20:00:02.824256 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.824231 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" event={"ID":"60c26d2d-3fa6-403e-9aa1-611dff153b14","Type":"ContainerDied","Data":"1ac04b2a63b29670efce4d05f8eb817a08bb499bb7cca41a540d142f361877f4"} Oct 31 20:00:02.824589 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.824494 6107 scope.go:122] "RemoveContainer" containerID="1ac04b2a63b29670efce4d05f8eb817a08bb499bb7cca41a540d142f361877f4" Oct 31 20:00:02.826426 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.826403 6107 generic.go:356] "Generic (PLEG): container finished" podID="40ed662e-cdac-4394-ac95-4b103b48bbff" containerID="b9827776365acf6bd3ae185ed1f61ec321dcc8095b90fd4349f8601ff98f0d30" exitCode=60 Oct 31 20:00:02.826591 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.826531 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" event={"ID":"40ed662e-cdac-4394-ac95-4b103b48bbff","Type":"ContainerDied","Data":"b9827776365acf6bd3ae185ed1f61ec321dcc8095b90fd4349f8601ff98f0d30"} Oct 31 20:00:02.827217 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.827094 6107 scope.go:122] "RemoveContainer" containerID="b9827776365acf6bd3ae185ed1f61ec321dcc8095b90fd4349f8601ff98f0d30" Oct 31 20:00:02.829038 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.828810 6107 generic.go:356] "Generic (PLEG): container finished" podID="d68b1ca1-2e79-4786-9799-94151203f153" containerID="7eba3b1f01f0537b07c8a348d66e2111934b30c400879aa4810f767a65cd9359" exitCode=60 Oct 31 20:00:02.829038 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.828948 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" event={"ID":"d68b1ca1-2e79-4786-9799-94151203f153","Type":"ContainerDied","Data":"7eba3b1f01f0537b07c8a348d66e2111934b30c400879aa4810f767a65cd9359"} Oct 31 20:00:02.829193 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.829095 6107 scope.go:122] "RemoveContainer" containerID="7eba3b1f01f0537b07c8a348d66e2111934b30c400879aa4810f767a65cd9359" Oct 31 20:00:02.831271 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.831250 6107 generic.go:356] "Generic (PLEG): container finished" podID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" containerID="0775c43e92b8e0940189acee9d7deed4ed40c5704ffe8b5654425ad641e961cc" exitCode=60 Oct 31 20:00:02.831371 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.831292 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" event={"ID":"3c3a32c2-060e-4568-9744-1dd49c0b4cba","Type":"ContainerDied","Data":"0775c43e92b8e0940189acee9d7deed4ed40c5704ffe8b5654425ad641e961cc"} Oct 31 20:00:02.831502 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.831487 6107 scope.go:122] "RemoveContainer" containerID="0775c43e92b8e0940189acee9d7deed4ed40c5704ffe8b5654425ad641e961cc" Oct 31 20:00:02.833041 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.833019 6107 generic.go:356] "Generic (PLEG): container finished" podID="620ec382-3553-4b53-8e0e-355469f9b0e5" containerID="09b7665a2af59ff29ed7b33424b0c7e0ef2fc6e4021d8a5afbc4597c916c7abc" exitCode=60 Oct 31 20:00:02.833133 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.833046 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" event={"ID":"620ec382-3553-4b53-8e0e-355469f9b0e5","Type":"ContainerDied","Data":"09b7665a2af59ff29ed7b33424b0c7e0ef2fc6e4021d8a5afbc4597c916c7abc"} Oct 31 20:00:02.833282 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.833265 6107 scope.go:122] "RemoveContainer" containerID="09b7665a2af59ff29ed7b33424b0c7e0ef2fc6e4021d8a5afbc4597c916c7abc" Oct 31 20:00:02.835958 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.835935 6107 generic.go:356] "Generic (PLEG): container finished" podID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" containerID="f07a80922df9173b7441eb4e02486332db60cec6184bf5ee90639ca9c0cc9601" exitCode=60 Oct 31 20:00:02.836075 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.835981 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" event={"ID":"ed0e177d-af3a-41e8-bcd8-e963493aa6ca","Type":"ContainerDied","Data":"f07a80922df9173b7441eb4e02486332db60cec6184bf5ee90639ca9c0cc9601"} Oct 31 20:00:02.836283 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:02.836264 6107 scope.go:122] "RemoveContainer" containerID="f07a80922df9173b7441eb4e02486332db60cec6184bf5ee90639ca9c0cc9601" Oct 31 20:00:03.880959 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:03.874956 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" event={"ID":"29dfb5f8-7c03-427a-aa3c-4181855ed5fa","Type":"ContainerStarted","Data":"80c43fb4c2d349474c6771bc20b9608856fa578a08236ffb6569f82727bb7404"} Oct 31 20:00:03.880959 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:03.880434 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" event={"ID":"60c26d2d-3fa6-403e-9aa1-611dff153b14","Type":"ContainerStarted","Data":"354c2ca90a2e72bb741f581329104243562b0e1c38ed3616864e86bb081b710d"} Oct 31 20:00:03.886910 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:03.886850 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" event={"ID":"40ed662e-cdac-4394-ac95-4b103b48bbff","Type":"ContainerStarted","Data":"2a454eab5f1d7cf2c532092221e8b6c9b38a66804c05117ed06b596aceb1484f"} Oct 31 20:00:03.893464 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:03.893437 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" event={"ID":"d68b1ca1-2e79-4786-9799-94151203f153","Type":"ContainerStarted","Data":"1de57cb3106bfcca4b1e20f5859e2e8ff3ff9c0dad2cddea8667573edaf82b8d"} Oct 31 20:00:03.896626 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:03.896590 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" event={"ID":"3c3a32c2-060e-4568-9744-1dd49c0b4cba","Type":"ContainerStarted","Data":"9429002d6eae31c3114db0bfef3b0db57994b259f0b4982d5a56c02e41b65f1a"} Oct 31 20:00:03.899959 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:03.899934 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" event={"ID":"620ec382-3553-4b53-8e0e-355469f9b0e5","Type":"ContainerStarted","Data":"ccdc73b2c7eec17aa44d58742661b998f484d4732d7f53d84aaf00864174d207"} Oct 31 20:00:03.907959 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:03.905707 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" event={"ID":"ed0e177d-af3a-41e8-bcd8-e963493aa6ca","Type":"ContainerStarted","Data":"949434c72a00c0377d6d4aca2165e85601b7161859604594556f8fda7353951e"} Oct 31 20:00:04.088766 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.088712 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" podStartSLOduration=4.088695427 podStartE2EDuration="4.088695427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.087091396 +0000 UTC m=+2575.720485307" watchObservedRunningTime="2026-10-31 20:00:04.088695427 +0000 UTC m=+2575.722089334" Oct 31 20:00:04.286393 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:04.286360 6107 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620ec382_3553_4b53_8e0e_355469f9b0e5.slice/crio-ccdc73b2c7eec17aa44d58742661b998f484d4732d7f53d84aaf00864174d207.scope\": RecentStats: unable to find data in memory cache]" Oct 31 20:00:04.509388 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.509339 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" podStartSLOduration=4.509324593 podStartE2EDuration="4.509324593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.507041606 +0000 UTC m=+2576.140435525" watchObservedRunningTime="2026-10-31 20:00:04.509324593 +0000 UTC m=+2576.142718501" Oct 31 20:00:04.916284 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.916200 6107 generic.go:356] "Generic (PLEG): container finished" podID="d68b1ca1-2e79-4786-9799-94151203f153" containerID="1de57cb3106bfcca4b1e20f5859e2e8ff3ff9c0dad2cddea8667573edaf82b8d" exitCode=60 Oct 31 20:00:04.916726 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.916283 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" event={"ID":"d68b1ca1-2e79-4786-9799-94151203f153","Type":"ContainerDied","Data":"1de57cb3106bfcca4b1e20f5859e2e8ff3ff9c0dad2cddea8667573edaf82b8d"} Oct 31 20:00:04.916726 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.916345 6107 scope.go:122] "RemoveContainer" containerID="7eba3b1f01f0537b07c8a348d66e2111934b30c400879aa4810f767a65cd9359" Oct 31 20:00:04.916726 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.916557 6107 scope.go:122] "RemoveContainer" containerID="1de57cb3106bfcca4b1e20f5859e2e8ff3ff9c0dad2cddea8667573edaf82b8d" Oct 31 20:00:04.916925 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:04.916862 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv_odh-ai-gateway-infra(d68b1ca1-2e79-4786-9799-94151203f153)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" podUID="d68b1ca1-2e79-4786-9799-94151203f153" Oct 31 20:00:04.918481 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.918457 6107 generic.go:356] "Generic (PLEG): container finished" podID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" containerID="9429002d6eae31c3114db0bfef3b0db57994b259f0b4982d5a56c02e41b65f1a" exitCode=60 Oct 31 20:00:04.918575 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.918500 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" event={"ID":"3c3a32c2-060e-4568-9744-1dd49c0b4cba","Type":"ContainerDied","Data":"9429002d6eae31c3114db0bfef3b0db57994b259f0b4982d5a56c02e41b65f1a"} Oct 31 20:00:04.918753 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.918740 6107 scope.go:122] "RemoveContainer" containerID="9429002d6eae31c3114db0bfef3b0db57994b259f0b4982d5a56c02e41b65f1a" Oct 31 20:00:04.919081 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:04.919061 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx_odh-ai-gateway-infra(3c3a32c2-060e-4568-9744-1dd49c0b4cba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" Oct 31 20:00:04.920336 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.920316 6107 generic.go:356] "Generic (PLEG): container finished" podID="620ec382-3553-4b53-8e0e-355469f9b0e5" containerID="ccdc73b2c7eec17aa44d58742661b998f484d4732d7f53d84aaf00864174d207" exitCode=60 Oct 31 20:00:04.920431 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.920395 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" event={"ID":"620ec382-3553-4b53-8e0e-355469f9b0e5","Type":"ContainerDied","Data":"ccdc73b2c7eec17aa44d58742661b998f484d4732d7f53d84aaf00864174d207"} Oct 31 20:00:04.920605 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.920587 6107 scope.go:122] "RemoveContainer" containerID="ccdc73b2c7eec17aa44d58742661b998f484d4732d7f53d84aaf00864174d207" Oct 31 20:00:04.920909 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:04.920873 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891280-mhpjv_odh-ai-gateway-infra(620ec382-3553-4b53-8e0e-355469f9b0e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" Oct 31 20:00:04.922419 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.922399 6107 generic.go:356] "Generic (PLEG): container finished" podID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" containerID="949434c72a00c0377d6d4aca2165e85601b7161859604594556f8fda7353951e" exitCode=60 Oct 31 20:00:04.922517 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.922477 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" event={"ID":"ed0e177d-af3a-41e8-bcd8-e963493aa6ca","Type":"ContainerDied","Data":"949434c72a00c0377d6d4aca2165e85601b7161859604594556f8fda7353951e"} Oct 31 20:00:04.922654 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.922639 6107 scope.go:122] "RemoveContainer" containerID="949434c72a00c0377d6d4aca2165e85601b7161859604594556f8fda7353951e" Oct 31 20:00:04.922926 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:04.922899 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh_odh-ai-gateway-infra(ed0e177d-af3a-41e8-bcd8-e963493aa6ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" Oct 31 20:00:04.924440 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.924419 6107 generic.go:356] "Generic (PLEG): container finished" podID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" containerID="80c43fb4c2d349474c6771bc20b9608856fa578a08236ffb6569f82727bb7404" exitCode=60 Oct 31 20:00:04.924566 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.924549 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" event={"ID":"29dfb5f8-7c03-427a-aa3c-4181855ed5fa","Type":"ContainerDied","Data":"80c43fb4c2d349474c6771bc20b9608856fa578a08236ffb6569f82727bb7404"} Oct 31 20:00:04.924716 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.924701 6107 scope.go:122] "RemoveContainer" containerID="80c43fb4c2d349474c6771bc20b9608856fa578a08236ffb6569f82727bb7404" Oct 31 20:00:04.924939 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:04.924916 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns_odh-ai-gateway-infra(29dfb5f8-7c03-427a-aa3c-4181855ed5fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" Oct 31 20:00:04.926622 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.926588 6107 generic.go:356] "Generic (PLEG): container finished" podID="60c26d2d-3fa6-403e-9aa1-611dff153b14" containerID="354c2ca90a2e72bb741f581329104243562b0e1c38ed3616864e86bb081b710d" exitCode=60 Oct 31 20:00:04.926723 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.926686 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" event={"ID":"60c26d2d-3fa6-403e-9aa1-611dff153b14","Type":"ContainerDied","Data":"354c2ca90a2e72bb741f581329104243562b0e1c38ed3616864e86bb081b710d"} Oct 31 20:00:04.926877 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.926862 6107 scope.go:122] "RemoveContainer" containerID="354c2ca90a2e72bb741f581329104243562b0e1c38ed3616864e86bb081b710d" Oct 31 20:00:04.927138 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:04.927110 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm_odh-ai-gateway-infra(60c26d2d-3fa6-403e-9aa1-611dff153b14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" Oct 31 20:00:04.928748 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.928729 6107 generic.go:356] "Generic (PLEG): container finished" podID="40ed662e-cdac-4394-ac95-4b103b48bbff" containerID="2a454eab5f1d7cf2c532092221e8b6c9b38a66804c05117ed06b596aceb1484f" exitCode=60 Oct 31 20:00:04.928842 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.928771 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" event={"ID":"40ed662e-cdac-4394-ac95-4b103b48bbff","Type":"ContainerDied","Data":"2a454eab5f1d7cf2c532092221e8b6c9b38a66804c05117ed06b596aceb1484f"} Oct 31 20:00:04.929151 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.929132 6107 scope.go:122] "RemoveContainer" containerID="2a454eab5f1d7cf2c532092221e8b6c9b38a66804c05117ed06b596aceb1484f" Oct 31 20:00:04.929382 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:04.929363 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc_odh-ai-gateway-infra(40ed662e-cdac-4394-ac95-4b103b48bbff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" Oct 31 20:00:04.931679 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.931663 6107 scope.go:122] "RemoveContainer" containerID="0775c43e92b8e0940189acee9d7deed4ed40c5704ffe8b5654425ad641e961cc" Oct 31 20:00:04.950744 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.950727 6107 scope.go:122] "RemoveContainer" containerID="09b7665a2af59ff29ed7b33424b0c7e0ef2fc6e4021d8a5afbc4597c916c7abc" Oct 31 20:00:04.964170 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.964148 6107 scope.go:122] "RemoveContainer" containerID="f07a80922df9173b7441eb4e02486332db60cec6184bf5ee90639ca9c0cc9601" Oct 31 20:00:04.977840 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.977818 6107 scope.go:122] "RemoveContainer" containerID="675854e35e3f2b674f0d05761aa304538aab68f571812518512d1e4b3c6293fd" Oct 31 20:00:04.991421 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:04.991400 6107 scope.go:122] "RemoveContainer" containerID="1ac04b2a63b29670efce4d05f8eb817a08bb499bb7cca41a540d142f361877f4" Oct 31 20:00:05.006356 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:05.006334 6107 scope.go:122] "RemoveContainer" containerID="b9827776365acf6bd3ae185ed1f61ec321dcc8095b90fd4349f8601ff98f0d30" Oct 31 20:00:05.941110 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:05.941083 6107 scope.go:122] "RemoveContainer" containerID="1de57cb3106bfcca4b1e20f5859e2e8ff3ff9c0dad2cddea8667573edaf82b8d" Oct 31 20:00:05.941605 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:05.941411 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv_odh-ai-gateway-infra(d68b1ca1-2e79-4786-9799-94151203f153)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" podUID="d68b1ca1-2e79-4786-9799-94151203f153" Oct 31 20:00:05.943002 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:05.942960 6107 scope.go:122] "RemoveContainer" containerID="9429002d6eae31c3114db0bfef3b0db57994b259f0b4982d5a56c02e41b65f1a" Oct 31 20:00:05.943317 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:05.943294 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx_odh-ai-gateway-infra(3c3a32c2-060e-4568-9744-1dd49c0b4cba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" Oct 31 20:00:05.944553 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:05.944529 6107 scope.go:122] "RemoveContainer" containerID="ccdc73b2c7eec17aa44d58742661b998f484d4732d7f53d84aaf00864174d207" Oct 31 20:00:05.944893 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:05.944849 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891280-mhpjv_odh-ai-gateway-infra(620ec382-3553-4b53-8e0e-355469f9b0e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" Oct 31 20:00:05.946240 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:05.946223 6107 scope.go:122] "RemoveContainer" containerID="949434c72a00c0377d6d4aca2165e85601b7161859604594556f8fda7353951e" Oct 31 20:00:05.946498 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:05.946475 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh_odh-ai-gateway-infra(ed0e177d-af3a-41e8-bcd8-e963493aa6ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" Oct 31 20:00:05.948049 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:05.948032 6107 scope.go:122] "RemoveContainer" containerID="80c43fb4c2d349474c6771bc20b9608856fa578a08236ffb6569f82727bb7404" Oct 31 20:00:05.948301 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:05.948280 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns_odh-ai-gateway-infra(29dfb5f8-7c03-427a-aa3c-4181855ed5fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" Oct 31 20:00:05.949851 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:05.949833 6107 scope.go:122] "RemoveContainer" containerID="354c2ca90a2e72bb741f581329104243562b0e1c38ed3616864e86bb081b710d" Oct 31 20:00:05.950120 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:05.950098 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm_odh-ai-gateway-infra(60c26d2d-3fa6-403e-9aa1-611dff153b14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" Oct 31 20:00:05.951743 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:05.951724 6107 scope.go:122] "RemoveContainer" containerID="2a454eab5f1d7cf2c532092221e8b6c9b38a66804c05117ed06b596aceb1484f" Oct 31 20:00:05.952034 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:05.952010 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc_odh-ai-gateway-infra(40ed662e-cdac-4394-ac95-4b103b48bbff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" Oct 31 20:00:08.449831 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.449790 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488"] Oct 31 20:00:08.456004 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.455983 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:08.506415 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.506386 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488"] Oct 31 20:00:08.546805 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.546776 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-13e9b4-serving-cert\"" Oct 31 20:00:08.558694 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.558666 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-metrics-certs\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:08.558822 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.558710 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-maas-api-tls\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:08.558822 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.558738 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97f4\" (UniqueName: \"kubernetes.io/projected/fa28e736-5f55-4166-9457-4390fce20842-kube-api-access-p97f4\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:08.558989 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.558967 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa28e736-5f55-4166-9457-4390fce20842-openshift-service-ca\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:08.660511 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.660472 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-metrics-certs\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:08.660675 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.660531 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-maas-api-tls\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:08.660675 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.660561 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97f4\" (UniqueName: \"kubernetes.io/projected/fa28e736-5f55-4166-9457-4390fce20842-kube-api-access-p97f4\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:08.660817 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:08.660686 6107 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-serving-cert: secret "maas-api-e2e-api-b-13e9b4-serving-cert" not found Oct 31 20:00:08.660817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.660731 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa28e736-5f55-4166-9457-4390fce20842-openshift-service-ca\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:08.660817 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:08.660774 6107 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-maas-api-tls podName:fa28e736-5f55-4166-9457-4390fce20842 nodeName:}" failed. No retries permitted until 2026-10-31 20:00:09.160752912 +0000 UTC m=+2580.794146815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-maas-api-tls") pod "maas-api-e2e-api-b-13e9b4-5559788fd-xr488" (UID: "fa28e736-5f55-4166-9457-4390fce20842") : secret "maas-api-e2e-api-b-13e9b4-serving-cert" not found Oct 31 20:00:08.661383 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.661350 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa28e736-5f55-4166-9457-4390fce20842-openshift-service-ca\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:08.663019 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.662996 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-metrics-certs\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:08.715109 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:08.715020 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97f4\" (UniqueName: \"kubernetes.io/projected/fa28e736-5f55-4166-9457-4390fce20842-kube-api-access-p97f4\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:09.170859 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:09.170810 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-maas-api-tls\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:09.173232 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:09.173203 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-maas-api-tls\") pod \"maas-api-e2e-api-b-13e9b4-5559788fd-xr488\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:09.370154 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:09.370126 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:09.754711 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:00:09.754680 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa28e736_5f55_4166_9457_4390fce20842.slice/crio-33c6d31043020afe529e890c4faf881d6b9f5f1ecd1605b741099292def48bf9 WatchSource:0}: Error finding container 33c6d31043020afe529e890c4faf881d6b9f5f1ecd1605b741099292def48bf9: Status 404 returned error can't find the container with id 33c6d31043020afe529e890c4faf881d6b9f5f1ecd1605b741099292def48bf9 Oct 31 20:00:09.795288 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:09.795260 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488"] Oct 31 20:00:09.988359 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:09.988324 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" event={"ID":"fa28e736-5f55-4166-9457-4390fce20842","Type":"ContainerStarted","Data":"33c6d31043020afe529e890c4faf881d6b9f5f1ecd1605b741099292def48bf9"} Oct 31 20:00:10.998695 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:10.998656 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" event={"ID":"fa28e736-5f55-4166-9457-4390fce20842","Type":"ContainerStarted","Data":"5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab"} Oct 31 20:00:10.999193 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:10.998745 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:11.107375 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:11.107264 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" podStartSLOduration=2.073728645 podStartE2EDuration="3.107246313s" totalImagesPullingTime="1.033517668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:11.103574853 +0000 UTC m=+2582.736968749" watchObservedRunningTime="2026-10-31 20:00:11.107246313 +0000 UTC m=+2582.740640222" Oct 31 20:00:13.819961 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:13.819926 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb"] Oct 31 20:00:13.933959 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:13.933926 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc"] Oct 31 20:00:15.480269 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:15.480241 6107 scope.go:122] "RemoveContainer" containerID="9429002d6eae31c3114db0bfef3b0db57994b259f0b4982d5a56c02e41b65f1a" Oct 31 20:00:16.050521 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:16.050484 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" event={"ID":"3c3a32c2-060e-4568-9744-1dd49c0b4cba","Type":"ContainerStarted","Data":"7f08cd6153918bdf9cf7835c401f235e0a6f654df6949008ef119a64c7fa59a1"} Oct 31 20:00:16.480250 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:16.480212 6107 scope.go:122] "RemoveContainer" containerID="1de57cb3106bfcca4b1e20f5859e2e8ff3ff9c0dad2cddea8667573edaf82b8d" Oct 31 20:00:17.015015 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:17.014985 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:00:17.060655 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:17.060616 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" event={"ID":"d68b1ca1-2e79-4786-9799-94151203f153","Type":"ContainerStarted","Data":"43bab7996f45365dda3b8d3cbb7ab71e48d0fb800a3eca7c4703a9ca7c4f6bcb"} Oct 31 20:00:17.062511 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:17.062488 6107 generic.go:356] "Generic (PLEG): container finished" podID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" containerID="7f08cd6153918bdf9cf7835c401f235e0a6f654df6949008ef119a64c7fa59a1" exitCode=60 Oct 31 20:00:17.062627 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:17.062555 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" event={"ID":"3c3a32c2-060e-4568-9744-1dd49c0b4cba","Type":"ContainerDied","Data":"7f08cd6153918bdf9cf7835c401f235e0a6f654df6949008ef119a64c7fa59a1"} Oct 31 20:00:17.062627 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:17.062606 6107 scope.go:122] "RemoveContainer" containerID="9429002d6eae31c3114db0bfef3b0db57994b259f0b4982d5a56c02e41b65f1a" Oct 31 20:00:17.062976 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:17.062962 6107 scope.go:122] "RemoveContainer" containerID="7f08cd6153918bdf9cf7835c401f235e0a6f654df6949008ef119a64c7fa59a1" Oct 31 20:00:17.063345 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:17.063323 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx_odh-ai-gateway-infra(3c3a32c2-060e-4568-9744-1dd49c0b4cba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" Oct 31 20:00:17.169996 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:17.169963 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx"] Oct 31 20:00:17.480158 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:17.480122 6107 scope.go:122] "RemoveContainer" containerID="949434c72a00c0377d6d4aca2165e85601b7161859604594556f8fda7353951e" Oct 31 20:00:17.480437 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:17.480420 6107 scope.go:122] "RemoveContainer" containerID="2a454eab5f1d7cf2c532092221e8b6c9b38a66804c05117ed06b596aceb1484f" Oct 31 20:00:18.072292 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.072216 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" event={"ID":"40ed662e-cdac-4394-ac95-4b103b48bbff","Type":"ContainerStarted","Data":"baf74fd50ea2bab513245d1d60699e02136894f70271ff52e93fa0d22e85d8ab"} Oct 31 20:00:18.074115 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.074092 6107 generic.go:356] "Generic (PLEG): container finished" podID="d68b1ca1-2e79-4786-9799-94151203f153" containerID="43bab7996f45365dda3b8d3cbb7ab71e48d0fb800a3eca7c4703a9ca7c4f6bcb" exitCode=60 Oct 31 20:00:18.074264 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.074163 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" event={"ID":"d68b1ca1-2e79-4786-9799-94151203f153","Type":"ContainerDied","Data":"43bab7996f45365dda3b8d3cbb7ab71e48d0fb800a3eca7c4703a9ca7c4f6bcb"} Oct 31 20:00:18.074264 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.074212 6107 scope.go:122] "RemoveContainer" containerID="1de57cb3106bfcca4b1e20f5859e2e8ff3ff9c0dad2cddea8667573edaf82b8d" Oct 31 20:00:18.074611 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.074584 6107 scope.go:122] "RemoveContainer" containerID="43bab7996f45365dda3b8d3cbb7ab71e48d0fb800a3eca7c4703a9ca7c4f6bcb" Oct 31 20:00:18.074978 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:18.074957 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv_odh-ai-gateway-infra(d68b1ca1-2e79-4786-9799-94151203f153)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" podUID="d68b1ca1-2e79-4786-9799-94151203f153" Oct 31 20:00:18.077949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.077922 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" event={"ID":"ed0e177d-af3a-41e8-bcd8-e963493aa6ca","Type":"ContainerStarted","Data":"ab85bf9406a5a586dcd472e883b95714218bd31417a45e7588f0f5e634089b27"} Oct 31 20:00:18.243708 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.243673 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv"] Oct 31 20:00:18.337566 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.337502 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" Oct 31 20:00:18.420014 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.419774 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3a32c2-060e-4568-9744-1dd49c0b4cba-kube-api-access-qb7f5\" (UniqueName: \"kubernetes.io/projected/3c3a32c2-060e-4568-9744-1dd49c0b4cba-kube-api-access-qb7f5\") pod \"3c3a32c2-060e-4568-9744-1dd49c0b4cba\" (UID: \"3c3a32c2-060e-4568-9744-1dd49c0b4cba\") " Oct 31 20:00:18.420014 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.419846 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3a32c2-060e-4568-9744-1dd49c0b4cba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3a32c2-060e-4568-9744-1dd49c0b4cba-maas-api-service-ca\") pod \"3c3a32c2-060e-4568-9744-1dd49c0b4cba\" (UID: \"3c3a32c2-060e-4568-9744-1dd49c0b4cba\") " Oct 31 20:00:18.420479 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.420451 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3a32c2-060e-4568-9744-1dd49c0b4cba-maas-api-service-ca" pod "3c3a32c2-060e-4568-9744-1dd49c0b4cba" (UID: "3c3a32c2-060e-4568-9744-1dd49c0b4cba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:18.420635 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.420620 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3a32c2-060e-4568-9744-1dd49c0b4cba-maas-api-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:18.422243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.422215 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3a32c2-060e-4568-9744-1dd49c0b4cba-kube-api-access-qb7f5" pod "3c3a32c2-060e-4568-9744-1dd49c0b4cba" (UID: "3c3a32c2-060e-4568-9744-1dd49c0b4cba"). InnerVolumeSpecName "kube-api-access-qb7f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:18.480437 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.480400 6107 scope.go:122] "RemoveContainer" containerID="354c2ca90a2e72bb741f581329104243562b0e1c38ed3616864e86bb081b710d" Oct 31 20:00:18.480689 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.480670 6107 scope.go:122] "RemoveContainer" containerID="ccdc73b2c7eec17aa44d58742661b998f484d4732d7f53d84aaf00864174d207" Oct 31 20:00:18.480778 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.480759 6107 scope.go:122] "RemoveContainer" containerID="80c43fb4c2d349474c6771bc20b9608856fa578a08236ffb6569f82727bb7404" Oct 31 20:00:18.521918 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.521853 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb7f5\" (UniqueName: \"kubernetes.io/projected/3c3a32c2-060e-4568-9744-1dd49c0b4cba-kube-api-access-qb7f5\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:18.821090 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.821041 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" podUID="75491f8d-9f81-4f0b-afbf-d2a63b7b6731" containerName="payload-processing" containerID="cri-o://275ff3fc0b6d002f89a3290056cf77343238cde248030626d1c10eee32c87a1e" gracePeriod=25 Oct 31 20:00:18.924375 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.924225 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp"] Oct 31 20:00:18.924535 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.924430 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" podUID="049cf604-6e9d-435d-8f6f-e141c1fb289b" containerName="maas-api" containerID="cri-o://d5e42914a208fc8a341de41bbcedfcc9a76a1b596628e4e36ab0e64ac812c90d" gracePeriod=30 Oct 31 20:00:18.943151 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:18.941200 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" podUID="81d92768-7aee-4fa2-9777-d13bcc073772" containerName="payload-pre-processing" containerID="cri-o://cdfdd120eae2104565a093f18475544c6b7dff9035ec7a73abbbadfc3aead4cd" gracePeriod=25 Oct 31 20:00:19.088926 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.086671 6107 generic.go:356] "Generic (PLEG): container finished" podID="049cf604-6e9d-435d-8f6f-e141c1fb289b" containerID="d5e42914a208fc8a341de41bbcedfcc9a76a1b596628e4e36ab0e64ac812c90d" exitCode=0 Oct 31 20:00:19.088926 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.086815 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" event={"ID":"049cf604-6e9d-435d-8f6f-e141c1fb289b","Type":"ContainerDied","Data":"d5e42914a208fc8a341de41bbcedfcc9a76a1b596628e4e36ab0e64ac812c90d"} Oct 31 20:00:19.088926 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.088334 6107 generic.go:356] "Generic (PLEG): container finished" podID="75491f8d-9f81-4f0b-afbf-d2a63b7b6731" containerID="275ff3fc0b6d002f89a3290056cf77343238cde248030626d1c10eee32c87a1e" exitCode=0 Oct 31 20:00:19.088926 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.088388 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" event={"ID":"75491f8d-9f81-4f0b-afbf-d2a63b7b6731","Type":"ContainerDied","Data":"275ff3fc0b6d002f89a3290056cf77343238cde248030626d1c10eee32c87a1e"} Oct 31 20:00:19.091206 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.090546 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" event={"ID":"60c26d2d-3fa6-403e-9aa1-611dff153b14","Type":"ContainerStarted","Data":"b8511cc13fd5f1ead63699acb43b8ce0d42607e06240abe2c6a7cb2dcc5828a3"} Oct 31 20:00:19.092836 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.092810 6107 generic.go:356] "Generic (PLEG): container finished" podID="81d92768-7aee-4fa2-9777-d13bcc073772" containerID="cdfdd120eae2104565a093f18475544c6b7dff9035ec7a73abbbadfc3aead4cd" exitCode=0 Oct 31 20:00:19.092983 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.092869 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" event={"ID":"81d92768-7aee-4fa2-9777-d13bcc073772","Type":"ContainerDied","Data":"cdfdd120eae2104565a093f18475544c6b7dff9035ec7a73abbbadfc3aead4cd"} Oct 31 20:00:19.096134 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.096108 6107 generic.go:356] "Generic (PLEG): container finished" podID="40ed662e-cdac-4394-ac95-4b103b48bbff" containerID="baf74fd50ea2bab513245d1d60699e02136894f70271ff52e93fa0d22e85d8ab" exitCode=60 Oct 31 20:00:19.096250 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.096202 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" event={"ID":"40ed662e-cdac-4394-ac95-4b103b48bbff","Type":"ContainerDied","Data":"baf74fd50ea2bab513245d1d60699e02136894f70271ff52e93fa0d22e85d8ab"} Oct 31 20:00:19.096250 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.096236 6107 scope.go:122] "RemoveContainer" containerID="2a454eab5f1d7cf2c532092221e8b6c9b38a66804c05117ed06b596aceb1484f" Oct 31 20:00:19.096591 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.096575 6107 scope.go:122] "RemoveContainer" containerID="baf74fd50ea2bab513245d1d60699e02136894f70271ff52e93fa0d22e85d8ab" Oct 31 20:00:19.096954 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:19.096873 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc_odh-ai-gateway-infra(40ed662e-cdac-4394-ac95-4b103b48bbff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" Oct 31 20:00:19.102296 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.102270 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" event={"ID":"3c3a32c2-060e-4568-9744-1dd49c0b4cba","Type":"ContainerDied","Data":"e126e38680933fab4163d58e831c45537d7e7d8bfe2acbd6fa1e3ba13643c2d7"} Oct 31 20:00:19.102384 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.102372 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx" Oct 31 20:00:19.114766 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.114277 6107 generic.go:356] "Generic (PLEG): container finished" podID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" containerID="ab85bf9406a5a586dcd472e883b95714218bd31417a45e7588f0f5e634089b27" exitCode=60 Oct 31 20:00:19.114766 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.114321 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" event={"ID":"ed0e177d-af3a-41e8-bcd8-e963493aa6ca","Type":"ContainerDied","Data":"ab85bf9406a5a586dcd472e883b95714218bd31417a45e7588f0f5e634089b27"} Oct 31 20:00:19.115259 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.115155 6107 scope.go:122] "RemoveContainer" containerID="ab85bf9406a5a586dcd472e883b95714218bd31417a45e7588f0f5e634089b27" Oct 31 20:00:19.115783 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:19.115727 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh_odh-ai-gateway-infra(ed0e177d-af3a-41e8-bcd8-e963493aa6ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" Oct 31 20:00:19.133849 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.133266 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc"] Oct 31 20:00:19.180256 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.180232 6107 scope.go:122] "RemoveContainer" containerID="7f08cd6153918bdf9cf7835c401f235e0a6f654df6949008ef119a64c7fa59a1" Oct 31 20:00:19.234198 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.234165 6107 scope.go:122] "RemoveContainer" containerID="949434c72a00c0377d6d4aca2165e85601b7161859604594556f8fda7353951e" Oct 31 20:00:19.308085 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.307645 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" Oct 31 20:00:19.368176 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.366424 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-plugins-config-volume\") pod \"75491f8d-9f81-4f0b-afbf-d2a63b7b6731\" (UID: \"75491f8d-9f81-4f0b-afbf-d2a63b7b6731\") " Oct 31 20:00:19.368176 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.366684 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-kube-api-access-tk9m5\" (UniqueName: \"kubernetes.io/projected/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-kube-api-access-tk9m5\") pod \"75491f8d-9f81-4f0b-afbf-d2a63b7b6731\" (UID: \"75491f8d-9f81-4f0b-afbf-d2a63b7b6731\") " Oct 31 20:00:19.383227 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.383135 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-plugins-config-volume" pod "75491f8d-9f81-4f0b-afbf-d2a63b7b6731" (UID: "75491f8d-9f81-4f0b-afbf-d2a63b7b6731"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:19.414675 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.414621 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-kube-api-access-tk9m5" pod "75491f8d-9f81-4f0b-afbf-d2a63b7b6731" (UID: "75491f8d-9f81-4f0b-afbf-d2a63b7b6731"). InnerVolumeSpecName "kube-api-access-tk9m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:19.421136 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.421106 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh"] Oct 31 20:00:19.475699 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.475564 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 20:00:19.476526 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.476498 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:19.476599 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.476531 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk9m5\" (UniqueName: \"kubernetes.io/projected/75491f8d-9f81-4f0b-afbf-d2a63b7b6731-kube-api-access-tk9m5\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:19.477769 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.477695 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" Oct 31 20:00:19.500207 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.500177 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" Oct 31 20:00:19.568564 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.568531 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx"] Oct 31 20:00:19.582987 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.582947 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-maas-api-tls\") pod \"049cf604-6e9d-435d-8f6f-e141c1fb289b\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " Oct 31 20:00:19.583167 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.583061 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049cf604-6e9d-435d-8f6f-e141c1fb289b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049cf604-6e9d-435d-8f6f-e141c1fb289b-openshift-service-ca\") pod \"049cf604-6e9d-435d-8f6f-e141c1fb289b\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " Oct 31 20:00:19.583167 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.583123 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d68b1ca1-2e79-4786-9799-94151203f153-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b1ca1-2e79-4786-9799-94151203f153-maas-api-service-ca\") pod \"d68b1ca1-2e79-4786-9799-94151203f153\" (UID: \"d68b1ca1-2e79-4786-9799-94151203f153\") " Oct 31 20:00:19.583263 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.583171 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049cf604-6e9d-435d-8f6f-e141c1fb289b-kube-api-access-cdlft\" (UniqueName: \"kubernetes.io/projected/049cf604-6e9d-435d-8f6f-e141c1fb289b-kube-api-access-cdlft\") pod \"049cf604-6e9d-435d-8f6f-e141c1fb289b\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " Oct 31 20:00:19.583309 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.583275 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d92768-7aee-4fa2-9777-d13bcc073772-kube-api-access-888lj\" (UniqueName: \"kubernetes.io/projected/81d92768-7aee-4fa2-9777-d13bcc073772-kube-api-access-888lj\") pod \"81d92768-7aee-4fa2-9777-d13bcc073772\" (UID: \"81d92768-7aee-4fa2-9777-d13bcc073772\") " Oct 31 20:00:19.583408 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.583390 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68b1ca1-2e79-4786-9799-94151203f153-kube-api-access-nt8vv\" (UniqueName: \"kubernetes.io/projected/d68b1ca1-2e79-4786-9799-94151203f153-kube-api-access-nt8vv\") pod \"d68b1ca1-2e79-4786-9799-94151203f153\" (UID: \"d68b1ca1-2e79-4786-9799-94151203f153\") " Oct 31 20:00:19.583455 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.583424 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-metrics-certs\") pod \"049cf604-6e9d-435d-8f6f-e141c1fb289b\" (UID: \"049cf604-6e9d-435d-8f6f-e141c1fb289b\") " Oct 31 20:00:19.583501 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.583463 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d92768-7aee-4fa2-9777-d13bcc073772-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d92768-7aee-4fa2-9777-d13bcc073772-plugins-config-volume\") pod \"81d92768-7aee-4fa2-9777-d13bcc073772\" (UID: \"81d92768-7aee-4fa2-9777-d13bcc073772\") " Oct 31 20:00:19.584566 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.584535 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7c0d34-29891280-f7zqx"] Oct 31 20:00:19.584720 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.584683 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d92768-7aee-4fa2-9777-d13bcc073772-plugins-config-volume" pod "81d92768-7aee-4fa2-9777-d13bcc073772" (UID: "81d92768-7aee-4fa2-9777-d13bcc073772"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:19.586559 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.586525 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68b1ca1-2e79-4786-9799-94151203f153-maas-api-service-ca" pod "d68b1ca1-2e79-4786-9799-94151203f153" (UID: "d68b1ca1-2e79-4786-9799-94151203f153"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:19.587388 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.587090 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049cf604-6e9d-435d-8f6f-e141c1fb289b-openshift-service-ca" pod "049cf604-6e9d-435d-8f6f-e141c1fb289b" (UID: "049cf604-6e9d-435d-8f6f-e141c1fb289b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:19.604436 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.601242 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d92768-7aee-4fa2-9777-d13bcc073772-kube-api-access-888lj" pod "81d92768-7aee-4fa2-9777-d13bcc073772" (UID: "81d92768-7aee-4fa2-9777-d13bcc073772"). InnerVolumeSpecName "kube-api-access-888lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:19.604436 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.601324 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049cf604-6e9d-435d-8f6f-e141c1fb289b-kube-api-access-cdlft" pod "049cf604-6e9d-435d-8f6f-e141c1fb289b" (UID: "049cf604-6e9d-435d-8f6f-e141c1fb289b"). InnerVolumeSpecName "kube-api-access-cdlft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:19.604436 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.601377 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-maas-api-tls" pod "049cf604-6e9d-435d-8f6f-e141c1fb289b" (UID: "049cf604-6e9d-435d-8f6f-e141c1fb289b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:00:19.604436 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.601423 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-metrics-certs" pod "049cf604-6e9d-435d-8f6f-e141c1fb289b" (UID: "049cf604-6e9d-435d-8f6f-e141c1fb289b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:00:19.605933 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.605699 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68b1ca1-2e79-4786-9799-94151203f153-kube-api-access-nt8vv" pod "d68b1ca1-2e79-4786-9799-94151203f153" (UID: "d68b1ca1-2e79-4786-9799-94151203f153"). InnerVolumeSpecName "kube-api-access-nt8vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:19.685772 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.685685 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-888lj\" (UniqueName: \"kubernetes.io/projected/81d92768-7aee-4fa2-9777-d13bcc073772-kube-api-access-888lj\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:19.685772 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.685714 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt8vv\" (UniqueName: \"kubernetes.io/projected/d68b1ca1-2e79-4786-9799-94151203f153-kube-api-access-nt8vv\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:19.685772 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.685725 6107 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-metrics-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:19.685772 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.685734 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d92768-7aee-4fa2-9777-d13bcc073772-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:19.685772 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.685744 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049cf604-6e9d-435d-8f6f-e141c1fb289b-maas-api-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:19.685772 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.685753 6107 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049cf604-6e9d-435d-8f6f-e141c1fb289b-openshift-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:19.685772 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.685762 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b1ca1-2e79-4786-9799-94151203f153-maas-api-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:19.685772 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:19.685770 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdlft\" (UniqueName: \"kubernetes.io/projected/049cf604-6e9d-435d-8f6f-e141c1fb289b-kube-api-access-cdlft\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:20.139494 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.139450 6107 generic.go:356] "Generic (PLEG): container finished" podID="620ec382-3553-4b53-8e0e-355469f9b0e5" containerID="9bb074d998d58c0fd3c4ac7e93c5b0ad55d626c0b1967e6e4ef6081e0e6158e8" exitCode=60 Oct 31 20:00:20.139978 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.139493 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" event={"ID":"620ec382-3553-4b53-8e0e-355469f9b0e5","Type":"ContainerDied","Data":"9bb074d998d58c0fd3c4ac7e93c5b0ad55d626c0b1967e6e4ef6081e0e6158e8"} Oct 31 20:00:20.139978 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.139741 6107 scope.go:122] "RemoveContainer" containerID="ccdc73b2c7eec17aa44d58742661b998f484d4732d7f53d84aaf00864174d207" Oct 31 20:00:20.140218 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.140197 6107 scope.go:122] "RemoveContainer" containerID="9bb074d998d58c0fd3c4ac7e93c5b0ad55d626c0b1967e6e4ef6081e0e6158e8" Oct 31 20:00:20.140633 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:20.140606 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891280-mhpjv_odh-ai-gateway-infra(620ec382-3553-4b53-8e0e-355469f9b0e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" Oct 31 20:00:20.150720 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.150466 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" event={"ID":"049cf604-6e9d-435d-8f6f-e141c1fb289b","Type":"ContainerDied","Data":"952bca6d11b5e76b5c6d96a907c4cc2f621a2f17ca4eb04ba54c1485913071ab"} Oct 31 20:00:20.150720 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.150592 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp" Oct 31 20:00:20.156115 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.156082 6107 generic.go:356] "Generic (PLEG): container finished" podID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" containerID="54490b04ccc0037a9ca99376db460669d10669fbbf8a4ae85e9b5111354a1a69" exitCode=60 Oct 31 20:00:20.156261 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.156238 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" event={"ID":"29dfb5f8-7c03-427a-aa3c-4181855ed5fa","Type":"ContainerDied","Data":"54490b04ccc0037a9ca99376db460669d10669fbbf8a4ae85e9b5111354a1a69"} Oct 31 20:00:20.156749 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.156727 6107 scope.go:122] "RemoveContainer" containerID="54490b04ccc0037a9ca99376db460669d10669fbbf8a4ae85e9b5111354a1a69" Oct 31 20:00:20.157165 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:20.157134 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns_odh-ai-gateway-infra(29dfb5f8-7c03-427a-aa3c-4181855ed5fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" Oct 31 20:00:20.162595 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.162575 6107 scope.go:122] "RemoveContainer" containerID="d5e42914a208fc8a341de41bbcedfcc9a76a1b596628e4e36ab0e64ac812c90d" Oct 31 20:00:20.163935 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.163911 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" event={"ID":"75491f8d-9f81-4f0b-afbf-d2a63b7b6731","Type":"ContainerDied","Data":"cf2857eab0b5311994a155b83658154ea4c3433fdc018d373ea3632a1758d997"} Oct 31 20:00:20.164070 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.164053 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb" Oct 31 20:00:20.167319 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.167297 6107 generic.go:356] "Generic (PLEG): container finished" podID="60c26d2d-3fa6-403e-9aa1-611dff153b14" containerID="b8511cc13fd5f1ead63699acb43b8ce0d42607e06240abe2c6a7cb2dcc5828a3" exitCode=60 Oct 31 20:00:20.167416 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.167366 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" event={"ID":"60c26d2d-3fa6-403e-9aa1-611dff153b14","Type":"ContainerDied","Data":"b8511cc13fd5f1ead63699acb43b8ce0d42607e06240abe2c6a7cb2dcc5828a3"} Oct 31 20:00:20.167762 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.167744 6107 scope.go:122] "RemoveContainer" containerID="b8511cc13fd5f1ead63699acb43b8ce0d42607e06240abe2c6a7cb2dcc5828a3" Oct 31 20:00:20.168101 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:20.168075 6107 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm_odh-ai-gateway-infra(60c26d2d-3fa6-403e-9aa1-611dff153b14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" Oct 31 20:00:20.169941 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.169499 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" event={"ID":"81d92768-7aee-4fa2-9777-d13bcc073772","Type":"ContainerDied","Data":"ab22d5e445186264e8641d3adf2c6633325c67e0ba92e5d108bed61b17d02219"} Oct 31 20:00:20.169941 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.169580 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc" Oct 31 20:00:20.179016 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.178994 6107 scope.go:122] "RemoveContainer" containerID="80c43fb4c2d349474c6771bc20b9608856fa578a08236ffb6569f82727bb7404" Oct 31 20:00:20.181744 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.181720 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" event={"ID":"d68b1ca1-2e79-4786-9799-94151203f153","Type":"ContainerDied","Data":"ae19e6589aeb70da7d58e9a8488d9918ccca69155218ea9c9d3489b2fe38e55f"} Oct 31 20:00:20.181855 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.181837 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv" Oct 31 20:00:20.200519 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.200494 6107 scope.go:122] "RemoveContainer" containerID="275ff3fc0b6d002f89a3290056cf77343238cde248030626d1c10eee32c87a1e" Oct 31 20:00:20.214318 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.214237 6107 scope.go:122] "RemoveContainer" containerID="354c2ca90a2e72bb741f581329104243562b0e1c38ed3616864e86bb081b710d" Oct 31 20:00:20.243451 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.242178 6107 scope.go:122] "RemoveContainer" containerID="cdfdd120eae2104565a093f18475544c6b7dff9035ec7a73abbbadfc3aead4cd" Oct 31 20:00:20.257384 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.254428 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm"] Oct 31 20:00:20.276356 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.276330 6107 scope.go:122] "RemoveContainer" containerID="43bab7996f45365dda3b8d3cbb7ab71e48d0fb800a3eca7c4703a9ca7c4f6bcb" Oct 31 20:00:20.380546 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.380519 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" Oct 31 20:00:20.406332 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:20.406290 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:00:20.422153 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.422122 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" Oct 31 20:00:20.496443 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.496410 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" path="/var/lib/kubelet/pods/3c3a32c2-060e-4568-9744-1dd49c0b4cba/volumes" Oct 31 20:00:20.496649 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.496421 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40ed662e-cdac-4394-ac95-4b103b48bbff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ed662e-cdac-4394-ac95-4b103b48bbff-maas-api-service-ca\") pod \"40ed662e-cdac-4394-ac95-4b103b48bbff\" (UID: \"40ed662e-cdac-4394-ac95-4b103b48bbff\") " Oct 31 20:00:20.496727 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.496666 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-maas-api-service-ca\") pod \"ed0e177d-af3a-41e8-bcd8-e963493aa6ca\" (UID: \"ed0e177d-af3a-41e8-bcd8-e963493aa6ca\") " Oct 31 20:00:20.496787 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.496737 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ed662e-cdac-4394-ac95-4b103b48bbff-kube-api-access-h868g\" (UniqueName: \"kubernetes.io/projected/40ed662e-cdac-4394-ac95-4b103b48bbff-kube-api-access-h868g\") pod \"40ed662e-cdac-4394-ac95-4b103b48bbff\" (UID: \"40ed662e-cdac-4394-ac95-4b103b48bbff\") " Oct 31 20:00:20.496837 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.496776 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ed662e-cdac-4394-ac95-4b103b48bbff-maas-api-service-ca" pod "40ed662e-cdac-4394-ac95-4b103b48bbff" (UID: "40ed662e-cdac-4394-ac95-4b103b48bbff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:20.496837 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.496819 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-kube-api-access-qwz7d\" (UniqueName: \"kubernetes.io/projected/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-kube-api-access-qwz7d\") pod \"ed0e177d-af3a-41e8-bcd8-e963493aa6ca\" (UID: \"ed0e177d-af3a-41e8-bcd8-e963493aa6ca\") " Oct 31 20:00:20.497098 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.497074 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-maas-api-service-ca" pod "ed0e177d-af3a-41e8-bcd8-e963493aa6ca" (UID: "ed0e177d-af3a-41e8-bcd8-e963493aa6ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:20.498095 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.497573 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ed662e-cdac-4394-ac95-4b103b48bbff-maas-api-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:20.498095 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.497607 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-maas-api-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:20.499142 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.499115 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ed662e-cdac-4394-ac95-4b103b48bbff-kube-api-access-h868g" pod "40ed662e-cdac-4394-ac95-4b103b48bbff" (UID: "40ed662e-cdac-4394-ac95-4b103b48bbff"). InnerVolumeSpecName "kube-api-access-h868g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:20.499419 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.499394 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-kube-api-access-qwz7d" pod "ed0e177d-af3a-41e8-bcd8-e963493aa6ca" (UID: "ed0e177d-af3a-41e8-bcd8-e963493aa6ca"). InnerVolumeSpecName "kube-api-access-qwz7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:20.598647 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.598616 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h868g\" (UniqueName: \"kubernetes.io/projected/40ed662e-cdac-4394-ac95-4b103b48bbff-kube-api-access-h868g\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:20.598803 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.598722 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwz7d\" (UniqueName: \"kubernetes.io/projected/ed0e177d-af3a-41e8-bcd8-e963493aa6ca-kube-api-access-qwz7d\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:20.669291 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.669176 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv"] Oct 31 20:00:20.673930 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.673899 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5370f5-29891280-zvdkv"] Oct 31 20:00:20.777171 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.777141 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp"] Oct 31 20:00:20.785020 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.784993 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9bbcf7-6946574d4b-797gp"] Oct 31 20:00:20.885557 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.885518 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb"] Oct 31 20:00:20.889578 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:20.889546 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9bbcf7-7dbd7ddd49-28btb"] Oct 31 20:00:21.097003 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.096963 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc"] Oct 31 20:00:21.100717 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.100688 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9bbcf7-6cb46989d7-z7jkc"] Oct 31 20:00:21.193036 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.193010 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" Oct 31 20:00:21.193036 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.193035 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh" event={"ID":"ed0e177d-af3a-41e8-bcd8-e963493aa6ca","Type":"ContainerDied","Data":"abab73c525800064f6d0de1e5cf2e03c20242c1eaad5ccd755b3bbfd0e37b466"} Oct 31 20:00:21.193562 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.193095 6107 scope.go:122] "RemoveContainer" containerID="ab85bf9406a5a586dcd472e883b95714218bd31417a45e7588f0f5e634089b27" Oct 31 20:00:21.251623 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.251586 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" event={"ID":"40ed662e-cdac-4394-ac95-4b103b48bbff","Type":"ContainerDied","Data":"5909b57b538e005046c910450f66ebe233f9a71faf3a17b823e60df6522401bf"} Oct 31 20:00:21.251752 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.251706 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc" Oct 31 20:00:21.259555 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.259527 6107 scope.go:122] "RemoveContainer" containerID="baf74fd50ea2bab513245d1d60699e02136894f70271ff52e93fa0d22e85d8ab" Oct 31 20:00:21.263025 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.262997 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv"] Oct 31 20:00:21.482332 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.482298 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" Oct 31 20:00:21.545235 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.545210 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" Oct 31 20:00:21.556387 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.556349 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh"] Oct 31 20:00:21.562233 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.562205 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-967dee-29891280-bcnwh"] Oct 31 20:00:21.644988 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.644873 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620ec382-3553-4b53-8e0e-355469f9b0e5-kube-api-access-wm8tc\" (UniqueName: \"kubernetes.io/projected/620ec382-3553-4b53-8e0e-355469f9b0e5-kube-api-access-wm8tc\") pod \"620ec382-3553-4b53-8e0e-355469f9b0e5\" (UID: \"620ec382-3553-4b53-8e0e-355469f9b0e5\") " Oct 31 20:00:21.645138 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.645035 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620ec382-3553-4b53-8e0e-355469f9b0e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620ec382-3553-4b53-8e0e-355469f9b0e5-maas-api-service-ca\") pod \"620ec382-3553-4b53-8e0e-355469f9b0e5\" (UID: \"620ec382-3553-4b53-8e0e-355469f9b0e5\") " Oct 31 20:00:21.645138 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.645105 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c26d2d-3fa6-403e-9aa1-611dff153b14-kube-api-access-x4phg\" (UniqueName: \"kubernetes.io/projected/60c26d2d-3fa6-403e-9aa1-611dff153b14-kube-api-access-x4phg\") pod \"60c26d2d-3fa6-403e-9aa1-611dff153b14\" (UID: \"60c26d2d-3fa6-403e-9aa1-611dff153b14\") " Oct 31 20:00:21.645138 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.645122 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c26d2d-3fa6-403e-9aa1-611dff153b14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c26d2d-3fa6-403e-9aa1-611dff153b14-maas-api-service-ca\") pod \"60c26d2d-3fa6-403e-9aa1-611dff153b14\" (UID: \"60c26d2d-3fa6-403e-9aa1-611dff153b14\") " Oct 31 20:00:21.645455 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.645429 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620ec382-3553-4b53-8e0e-355469f9b0e5-maas-api-service-ca" pod "620ec382-3553-4b53-8e0e-355469f9b0e5" (UID: "620ec382-3553-4b53-8e0e-355469f9b0e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:21.645554 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.645490 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c26d2d-3fa6-403e-9aa1-611dff153b14-maas-api-service-ca" pod "60c26d2d-3fa6-403e-9aa1-611dff153b14" (UID: "60c26d2d-3fa6-403e-9aa1-611dff153b14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:21.646368 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.646344 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620ec382-3553-4b53-8e0e-355469f9b0e5-maas-api-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:21.646368 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.646369 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c26d2d-3fa6-403e-9aa1-611dff153b14-maas-api-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:21.647292 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.647262 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620ec382-3553-4b53-8e0e-355469f9b0e5-kube-api-access-wm8tc" pod "620ec382-3553-4b53-8e0e-355469f9b0e5" (UID: "620ec382-3553-4b53-8e0e-355469f9b0e5"). InnerVolumeSpecName "kube-api-access-wm8tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:21.647534 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.647511 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c26d2d-3fa6-403e-9aa1-611dff153b14-kube-api-access-x4phg" pod "60c26d2d-3fa6-403e-9aa1-611dff153b14" (UID: "60c26d2d-3fa6-403e-9aa1-611dff153b14"). InnerVolumeSpecName "kube-api-access-x4phg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:21.748322 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.748286 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm8tc\" (UniqueName: \"kubernetes.io/projected/620ec382-3553-4b53-8e0e-355469f9b0e5-kube-api-access-wm8tc\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:21.748322 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.748322 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4phg\" (UniqueName: \"kubernetes.io/projected/60c26d2d-3fa6-403e-9aa1-611dff153b14-kube-api-access-x4phg\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:21.772962 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.772930 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc"] Oct 31 20:00:21.776843 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:21.776811 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9bbcf7-29891280-kntpc"] Oct 31 20:00:22.003258 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.003221 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns"] Oct 31 20:00:22.270775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.270674 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" event={"ID":"29dfb5f8-7c03-427a-aa3c-4181855ed5fa","Type":"ContainerDied","Data":"928cd943075f762ef120195941edce924de7b5f3b50b8376c0447ef2cd6de0ec"} Oct 31 20:00:22.270775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.270741 6107 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="928cd943075f762ef120195941edce924de7b5f3b50b8376c0447ef2cd6de0ec" Oct 31 20:00:22.273323 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.273280 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" event={"ID":"60c26d2d-3fa6-403e-9aa1-611dff153b14","Type":"ContainerDied","Data":"9e46e5f5f219cd1aad102c1c438b2551b8b9d583532916897e1bd1093ce10000"} Oct 31 20:00:22.273569 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.273542 6107 scope.go:122] "RemoveContainer" containerID="b8511cc13fd5f1ead63699acb43b8ce0d42607e06240abe2c6a7cb2dcc5828a3" Oct 31 20:00:22.273704 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.273411 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm" Oct 31 20:00:22.277416 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.277282 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" Oct 31 20:00:22.277416 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.277338 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv" event={"ID":"620ec382-3553-4b53-8e0e-355469f9b0e5","Type":"ContainerDied","Data":"6abd6e334b1691350f60a51258ef71ae09c87cffe088e6defba54a9bbcb95c08"} Oct 31 20:00:22.278812 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.278792 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" Oct 31 20:00:22.286780 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.286763 6107 scope.go:122] "RemoveContainer" containerID="9bb074d998d58c0fd3c4ac7e93c5b0ad55d626c0b1967e6e4ef6081e0e6158e8" Oct 31 20:00:22.358269 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.358238 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-kube-api-access-6hgnr\" (UniqueName: \"kubernetes.io/projected/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-kube-api-access-6hgnr\") pod \"29dfb5f8-7c03-427a-aa3c-4181855ed5fa\" (UID: \"29dfb5f8-7c03-427a-aa3c-4181855ed5fa\") " Oct 31 20:00:22.358430 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.358395 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-maas-api-service-ca\") pod \"29dfb5f8-7c03-427a-aa3c-4181855ed5fa\" (UID: \"29dfb5f8-7c03-427a-aa3c-4181855ed5fa\") " Oct 31 20:00:22.358760 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.358737 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-maas-api-service-ca" pod "29dfb5f8-7c03-427a-aa3c-4181855ed5fa" (UID: "29dfb5f8-7c03-427a-aa3c-4181855ed5fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:22.359104 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.359084 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-maas-api-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:22.360365 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.360340 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-kube-api-access-6hgnr" pod "29dfb5f8-7c03-427a-aa3c-4181855ed5fa" (UID: "29dfb5f8-7c03-427a-aa3c-4181855ed5fa"). InnerVolumeSpecName "kube-api-access-6hgnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:22.429246 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.429211 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv"] Oct 31 20:00:22.433848 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.433819 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-mhpjv"] Oct 31 20:00:22.460239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.460212 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hgnr\" (UniqueName: \"kubernetes.io/projected/29dfb5f8-7c03-427a-aa3c-4181855ed5fa-kube-api-access-6hgnr\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:22.488985 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.488953 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049cf604-6e9d-435d-8f6f-e141c1fb289b" path="/var/lib/kubelet/pods/049cf604-6e9d-435d-8f6f-e141c1fb289b/volumes" Oct 31 20:00:22.489490 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.489470 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" path="/var/lib/kubelet/pods/40ed662e-cdac-4394-ac95-4b103b48bbff/volumes" Oct 31 20:00:22.490003 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.489984 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" path="/var/lib/kubelet/pods/620ec382-3553-4b53-8e0e-355469f9b0e5/volumes" Oct 31 20:00:22.490733 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.490710 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75491f8d-9f81-4f0b-afbf-d2a63b7b6731" path="/var/lib/kubelet/pods/75491f8d-9f81-4f0b-afbf-d2a63b7b6731/volumes" Oct 31 20:00:22.491199 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.491180 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d92768-7aee-4fa2-9777-d13bcc073772" path="/var/lib/kubelet/pods/81d92768-7aee-4fa2-9777-d13bcc073772/volumes" Oct 31 20:00:22.491762 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.491738 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d68b1ca1-2e79-4786-9799-94151203f153" path="/var/lib/kubelet/pods/d68b1ca1-2e79-4786-9799-94151203f153/volumes" Oct 31 20:00:22.492592 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.492570 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" path="/var/lib/kubelet/pods/ed0e177d-af3a-41e8-bcd8-e963493aa6ca/volumes" Oct 31 20:00:22.639295 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.639216 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm"] Oct 31 20:00:22.648913 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:22.648872 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-093b97-29891280-k6vbm"] Oct 31 20:00:23.288245 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:23.288218 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns" Oct 31 20:00:23.445106 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:23.445071 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns"] Oct 31 20:00:23.488853 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:23.488820 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b5155c-29891280-n4xns"] Oct 31 20:00:24.488827 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:24.488790 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" path="/var/lib/kubelet/pods/29dfb5f8-7c03-427a-aa3c-4181855ed5fa/volumes" Oct 31 20:00:24.489340 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:24.489323 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" path="/var/lib/kubelet/pods/60c26d2d-3fa6-403e-9aa1-611dff153b14/volumes" Oct 31 20:00:28.178014 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:28.177976 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx"] Oct 31 20:00:28.178522 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:28.178238 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" containerID="cri-o://5c59dfc0ec2509b4f8d009e1c9dfec5f2a6ab62b6141319f0ce982fb42fb3573" gracePeriod=30 Oct 31 20:00:33.395273 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.395236 6107 generic.go:356] "Generic (PLEG): container finished" podID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerID="5c59dfc0ec2509b4f8d009e1c9dfec5f2a6ab62b6141319f0ce982fb42fb3573" exitCode=0 Oct 31 20:00:33.395708 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.395319 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" event={"ID":"63dee691-27c6-4a5f-882c-fc3ed46cbeb1","Type":"ContainerDied","Data":"5c59dfc0ec2509b4f8d009e1c9dfec5f2a6ab62b6141319f0ce982fb42fb3573"} Oct 31 20:00:33.395708 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.395413 6107 scope.go:122] "RemoveContainer" containerID="aeb1ffc08c610db47e0d29ac08780c55190d5f1a65d0bc52e67af92521a5c81c" Oct 31 20:00:33.471687 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.471661 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 20:00:33.527131 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527100 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-certs\") pod \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " Oct 31 20:00:33.527337 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527151 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-credential-socket\") pod \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " Oct 31 20:00:33.527337 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527194 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-envoy\") pod \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " Oct 31 20:00:33.527337 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527237 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-kube-api-access-l97qp\" (UniqueName: \"kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-kube-api-access-l97qp\") pod \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " Oct 31 20:00:33.527337 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527313 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-socket\") pod \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " Oct 31 20:00:33.527561 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527412 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-certs" pod "63dee691-27c6-4a5f-882c-fc3ed46cbeb1" (UID: "63dee691-27c6-4a5f-882c-fc3ed46cbeb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:00:33.527561 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527430 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-token\" (UniqueName: \"kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-token\") pod \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " Oct 31 20:00:33.527561 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527467 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-data\") pod \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " Oct 31 20:00:33.527561 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527502 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-podinfo\") pod \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " Oct 31 20:00:33.527561 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527519 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-credential-socket" pod "63dee691-27c6-4a5f-882c-fc3ed46cbeb1" (UID: "63dee691-27c6-4a5f-882c-fc3ed46cbeb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:00:33.527811 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527575 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istiod-ca-cert\") pod \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\" (UID: \"63dee691-27c6-4a5f-882c-fc3ed46cbeb1\") " Oct 31 20:00:33.527811 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527586 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-socket" pod "63dee691-27c6-4a5f-882c-fc3ed46cbeb1" (UID: "63dee691-27c6-4a5f-882c-fc3ed46cbeb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:00:33.527811 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527786 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-data" pod "63dee691-27c6-4a5f-882c-fc3ed46cbeb1" (UID: "63dee691-27c6-4a5f-882c-fc3ed46cbeb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:00:33.527995 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.527941 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istiod-ca-cert" pod "63dee691-27c6-4a5f-882c-fc3ed46cbeb1" (UID: "63dee691-27c6-4a5f-882c-fc3ed46cbeb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:33.528346 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.528323 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:33.528407 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.528355 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:33.528407 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.528371 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:33.528407 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.528386 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:33.528407 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.528400 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:33.529492 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.529471 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-envoy" pod "63dee691-27c6-4a5f-882c-fc3ed46cbeb1" (UID: "63dee691-27c6-4a5f-882c-fc3ed46cbeb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:00:33.529609 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.529519 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-token" pod "63dee691-27c6-4a5f-882c-fc3ed46cbeb1" (UID: "63dee691-27c6-4a5f-882c-fc3ed46cbeb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:33.529824 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.529799 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-podinfo" pod "63dee691-27c6-4a5f-882c-fc3ed46cbeb1" (UID: "63dee691-27c6-4a5f-882c-fc3ed46cbeb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:00:33.529824 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.529818 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-kube-api-access-l97qp" pod "63dee691-27c6-4a5f-882c-fc3ed46cbeb1" (UID: "63dee691-27c6-4a5f-882c-fc3ed46cbeb1"). InnerVolumeSpecName "kube-api-access-l97qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:33.630357 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.630328 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:33.630357 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.630357 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l97qp\" (UniqueName: \"kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-kube-api-access-l97qp\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:33.630613 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.630370 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:33.630613 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:33.630379 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63dee691-27c6-4a5f-882c-fc3ed46cbeb1-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:34.404804 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:34.404773 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" Oct 31 20:00:34.404804 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:34.404797 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx" event={"ID":"63dee691-27c6-4a5f-882c-fc3ed46cbeb1","Type":"ContainerDied","Data":"3fcae3a00e22db7bd973db1a7d82778f95f7eb746fd90a869279861c0bcb7cd0"} Oct 31 20:00:34.405346 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:34.404911 6107 scope.go:122] "RemoveContainer" containerID="5c59dfc0ec2509b4f8d009e1c9dfec5f2a6ab62b6141319f0ce982fb42fb3573" Oct 31 20:00:34.477677 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:34.477647 6107 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63dee691_27c6_4a5f_882c_fc3ed46cbeb1.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:00:34.604368 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:34.604327 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx"] Oct 31 20:00:34.650184 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:34.650149 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9bbcf7-openshift-default-57fd5b8bcf-4bgbx"] Oct 31 20:00:36.489500 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:36.489465 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" path="/var/lib/kubelet/pods/63dee691-27c6-4a5f-882c-fc3ed46cbeb1/volumes" Oct 31 20:00:50.429990 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:00:50.429866 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:00:57.543922 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:57.543859 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5"] Oct 31 20:00:57.656508 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:57.656473 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6"] Oct 31 20:00:58.621531 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:58.621493 6107 generic.go:356] "Generic (PLEG): container finished" podID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerID="ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273" exitCode=0 Oct 31 20:00:58.622031 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:58.621548 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" event={"ID":"d2824e53-a817-4149-b6e4-49453b7c3d5b","Type":"ContainerDied","Data":"ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273"} Oct 31 20:00:58.622086 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:58.622071 6107 scope.go:122] "RemoveContainer" containerID="ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273" Oct 31 20:00:58.624258 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:58.624227 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:00:58.624348 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:58.624332 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:00:58.624395 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:58.624380 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:00:59.647526 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:00:59.647488 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" event={"ID":"d2824e53-a817-4149-b6e4-49453b7c3d5b","Type":"ContainerStarted","Data":"b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f"} Oct 31 20:01:00.370495 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:00.370452 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 20:01:00.370790 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:00.370765 6107 patch_prober.go:29] interesting pod/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 20:01:00.370857 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:00.370833 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 20:01:01.371171 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:01.371128 6107 patch_prober.go:29] interesting pod/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 20:01:01.371699 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:01.371208 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 20:01:02.371366 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:02.371327 6107 patch_prober.go:29] interesting pod/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 20:01:02.371811 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:02.371386 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 20:01:02.544579 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:02.544432 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" podUID="f7d03485-903b-480b-a923-08a9fd632e7b" containerName="payload-processing" containerID="cri-o://5cbafebe41170b87999632cba882bcbfc27b5c1a00f53b5f73996394913b71ba" gracePeriod=25 Oct 31 20:01:02.657753 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:02.657659 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" podUID="c17338a4-ac0b-4b76-a622-28fe74269fa9" containerName="payload-pre-processing" containerID="cri-o://54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878" gracePeriod=25 Oct 31 20:01:02.674935 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:02.674903 6107 generic.go:356] "Generic (PLEG): container finished" podID="f7d03485-903b-480b-a923-08a9fd632e7b" containerID="5cbafebe41170b87999632cba882bcbfc27b5c1a00f53b5f73996394913b71ba" exitCode=0 Oct 31 20:01:02.675100 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:02.674957 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" event={"ID":"f7d03485-903b-480b-a923-08a9fd632e7b","Type":"ContainerDied","Data":"5cbafebe41170b87999632cba882bcbfc27b5c1a00f53b5f73996394913b71ba"} Oct 31 20:01:02.889295 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:02.889263 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" Oct 31 20:01:03.051018 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.042558 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d03485-903b-480b-a923-08a9fd632e7b-kube-api-access-7hpk8\" (UniqueName: \"kubernetes.io/projected/f7d03485-903b-480b-a923-08a9fd632e7b-kube-api-access-7hpk8\") pod \"f7d03485-903b-480b-a923-08a9fd632e7b\" (UID: \"f7d03485-903b-480b-a923-08a9fd632e7b\") " Oct 31 20:01:03.051018 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.042852 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d03485-903b-480b-a923-08a9fd632e7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d03485-903b-480b-a923-08a9fd632e7b-plugins-config-volume\") pod \"f7d03485-903b-480b-a923-08a9fd632e7b\" (UID: \"f7d03485-903b-480b-a923-08a9fd632e7b\") " Oct 31 20:01:03.051018 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.043972 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d03485-903b-480b-a923-08a9fd632e7b-plugins-config-volume" pod "f7d03485-903b-480b-a923-08a9fd632e7b" (UID: "f7d03485-903b-480b-a923-08a9fd632e7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:03.051327 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.051150 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d03485-903b-480b-a923-08a9fd632e7b-kube-api-access-7hpk8" pod "f7d03485-903b-480b-a923-08a9fd632e7b" (UID: "f7d03485-903b-480b-a923-08a9fd632e7b"). InnerVolumeSpecName "kube-api-access-7hpk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:03.053042 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.052685 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" Oct 31 20:01:03.144964 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.144925 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c17338a4-ac0b-4b76-a622-28fe74269fa9-kube-api-access-s2gsb\" (UniqueName: \"kubernetes.io/projected/c17338a4-ac0b-4b76-a622-28fe74269fa9-kube-api-access-s2gsb\") pod \"c17338a4-ac0b-4b76-a622-28fe74269fa9\" (UID: \"c17338a4-ac0b-4b76-a622-28fe74269fa9\") " Oct 31 20:01:03.145163 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.144991 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c17338a4-ac0b-4b76-a622-28fe74269fa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17338a4-ac0b-4b76-a622-28fe74269fa9-plugins-config-volume\") pod \"c17338a4-ac0b-4b76-a622-28fe74269fa9\" (UID: \"c17338a4-ac0b-4b76-a622-28fe74269fa9\") " Oct 31 20:01:03.145376 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.145354 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17338a4-ac0b-4b76-a622-28fe74269fa9-plugins-config-volume" pod "c17338a4-ac0b-4b76-a622-28fe74269fa9" (UID: "c17338a4-ac0b-4b76-a622-28fe74269fa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:03.145565 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.145550 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17338a4-ac0b-4b76-a622-28fe74269fa9-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:03.145657 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.145567 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hpk8\" (UniqueName: \"kubernetes.io/projected/f7d03485-903b-480b-a923-08a9fd632e7b-kube-api-access-7hpk8\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:03.145657 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.145580 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d03485-903b-480b-a923-08a9fd632e7b-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:03.147206 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.147184 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17338a4-ac0b-4b76-a622-28fe74269fa9-kube-api-access-s2gsb" pod "c17338a4-ac0b-4b76-a622-28fe74269fa9" (UID: "c17338a4-ac0b-4b76-a622-28fe74269fa9"). InnerVolumeSpecName "kube-api-access-s2gsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:03.246570 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.246483 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2gsb\" (UniqueName: \"kubernetes.io/projected/c17338a4-ac0b-4b76-a622-28fe74269fa9-kube-api-access-s2gsb\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:03.370610 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.370575 6107 patch_prober.go:29] interesting pod/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 20:01:03.370784 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.370630 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 20:01:03.683579 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.683546 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" Oct 31 20:01:03.683579 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.683566 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5" event={"ID":"f7d03485-903b-480b-a923-08a9fd632e7b","Type":"ContainerDied","Data":"0ad8adbaf6b4ff768891c74f1a1809de290c4343913178719e7db009540ff271"} Oct 31 20:01:03.684133 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.683633 6107 scope.go:122] "RemoveContainer" containerID="5cbafebe41170b87999632cba882bcbfc27b5c1a00f53b5f73996394913b71ba" Oct 31 20:01:03.685288 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.685259 6107 generic.go:356] "Generic (PLEG): container finished" podID="c17338a4-ac0b-4b76-a622-28fe74269fa9" containerID="54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878" exitCode=0 Oct 31 20:01:03.685406 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.685291 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" event={"ID":"c17338a4-ac0b-4b76-a622-28fe74269fa9","Type":"ContainerDied","Data":"54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878"} Oct 31 20:01:03.685406 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.685313 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" Oct 31 20:01:03.685406 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.685349 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6" event={"ID":"c17338a4-ac0b-4b76-a622-28fe74269fa9","Type":"ContainerDied","Data":"73f387952d794ef4161f9f0ac9d1799acf7c366019c25bd36d95193272b835eb"} Oct 31 20:01:03.734252 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.734231 6107 scope.go:122] "RemoveContainer" containerID="54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878" Oct 31 20:01:03.748051 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.748030 6107 scope.go:122] "RemoveContainer" containerID="54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878" Oct 31 20:01:03.748343 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:01:03.748323 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878\": container with ID starting with 54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878 not found: ID does not exist" containerID="54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878" Oct 31 20:01:03.748420 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.748351 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878"} err="failed to get container status \"54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878\": rpc error: code = NotFound desc = could not find container \"54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878\": container with ID starting with 54b904ccda9966f66c0bc7fba5d660ea073434555264aeae4b522c1bc4df4878 not found: ID does not exist" Oct 31 20:01:03.840109 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.840027 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5"] Oct 31 20:01:03.884417 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.884385 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9a8346-75b479c5b9-rndb5"] Oct 31 20:01:03.985674 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.985601 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6"] Oct 31 20:01:03.991954 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:03.991923 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9a8346-7944cd8879-lhql6"] Oct 31 20:01:04.371060 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:04.371024 6107 patch_prober.go:29] interesting pod/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 20:01:04.371264 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:04.371089 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 20:01:04.543904 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:04.543131 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c17338a4-ac0b-4b76-a622-28fe74269fa9" path="/var/lib/kubelet/pods/c17338a4-ac0b-4b76-a622-28fe74269fa9/volumes" Oct 31 20:01:04.543904 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:04.543583 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d03485-903b-480b-a923-08a9fd632e7b" path="/var/lib/kubelet/pods/f7d03485-903b-480b-a923-08a9fd632e7b/volumes" Oct 31 20:01:05.376053 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:05.376020 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 20:01:05.376434 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:05.376382 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 20:01:05.377204 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:05.377187 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 20:01:17.288068 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:17.288020 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk"] Oct 31 20:01:17.401746 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:17.401706 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm"] Oct 31 20:01:19.320123 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.320081 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522"] Oct 31 20:01:19.320551 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.320332 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" podUID="63d0bee6-b082-4ab9-af16-d903398f9696" containerName="maas-api" containerID="cri-o://7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1" gracePeriod=30 Oct 31 20:01:19.634764 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.634736 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 20:01:19.791983 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.791938 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d0bee6-b082-4ab9-af16-d903398f9696-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d0bee6-b082-4ab9-af16-d903398f9696-openshift-service-ca\") pod \"63d0bee6-b082-4ab9-af16-d903398f9696\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " Oct 31 20:01:19.792188 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.792010 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-maas-api-tls\") pod \"63d0bee6-b082-4ab9-af16-d903398f9696\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " Oct 31 20:01:19.792267 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.792138 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-metrics-certs\") pod \"63d0bee6-b082-4ab9-af16-d903398f9696\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " Oct 31 20:01:19.792383 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.792365 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d0bee6-b082-4ab9-af16-d903398f9696-kube-api-access-xn2bd\" (UniqueName: \"kubernetes.io/projected/63d0bee6-b082-4ab9-af16-d903398f9696-kube-api-access-xn2bd\") pod \"63d0bee6-b082-4ab9-af16-d903398f9696\" (UID: \"63d0bee6-b082-4ab9-af16-d903398f9696\") " Oct 31 20:01:19.795752 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.795720 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-maas-api-tls" pod "63d0bee6-b082-4ab9-af16-d903398f9696" (UID: "63d0bee6-b082-4ab9-af16-d903398f9696"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:01:19.796031 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.796012 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d0bee6-b082-4ab9-af16-d903398f9696-openshift-service-ca" pod "63d0bee6-b082-4ab9-af16-d903398f9696" (UID: "63d0bee6-b082-4ab9-af16-d903398f9696"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:19.804919 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.804870 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d0bee6-b082-4ab9-af16-d903398f9696-kube-api-access-xn2bd" pod "63d0bee6-b082-4ab9-af16-d903398f9696" (UID: "63d0bee6-b082-4ab9-af16-d903398f9696"). InnerVolumeSpecName "kube-api-access-xn2bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:19.809688 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.809586 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-metrics-certs" pod "63d0bee6-b082-4ab9-af16-d903398f9696" (UID: "63d0bee6-b082-4ab9-af16-d903398f9696"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:01:19.823229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.823193 6107 generic.go:356] "Generic (PLEG): container finished" podID="63d0bee6-b082-4ab9-af16-d903398f9696" containerID="7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1" exitCode=0 Oct 31 20:01:19.823367 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.823300 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" event={"ID":"63d0bee6-b082-4ab9-af16-d903398f9696","Type":"ContainerDied","Data":"7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1"} Oct 31 20:01:19.823367 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.823353 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" event={"ID":"63d0bee6-b082-4ab9-af16-d903398f9696","Type":"ContainerDied","Data":"f5dacde436e55507c458964cfde6c1a961cca6d5bc742cfb344d95d6e16ef58c"} Oct 31 20:01:19.823487 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.823382 6107 scope.go:122] "RemoveContainer" containerID="7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1" Oct 31 20:01:19.823660 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.823640 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522" Oct 31 20:01:19.847930 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.847909 6107 scope.go:122] "RemoveContainer" containerID="7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1" Oct 31 20:01:19.848166 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:01:19.848148 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1\": container with ID starting with 7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1 not found: ID does not exist" containerID="7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1" Oct 31 20:01:19.848238 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.848173 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1"} err="failed to get container status \"7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1\": rpc error: code = NotFound desc = could not find container \"7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1\": container with ID starting with 7311f5f45b9e12d2d66192cb96cc1f3d71a2d252296cc2e3466b70db0d987db1 not found: ID does not exist" Oct 31 20:01:19.894316 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.894274 6107 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d0bee6-b082-4ab9-af16-d903398f9696-openshift-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:19.894316 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.894306 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-maas-api-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:19.894316 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.894317 6107 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d0bee6-b082-4ab9-af16-d903398f9696-metrics-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:19.894653 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.894330 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn2bd\" (UniqueName: \"kubernetes.io/projected/63d0bee6-b082-4ab9-af16-d903398f9696-kube-api-access-xn2bd\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:19.981457 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.981423 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522"] Oct 31 20:01:19.985099 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:19.985072 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dfe658-6585c8dfb9-59522"] Oct 31 20:01:20.405336 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:01:20.405296 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:01:20.488246 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:20.488212 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d0bee6-b082-4ab9-af16-d903398f9696" path="/var/lib/kubelet/pods/63d0bee6-b082-4ab9-af16-d903398f9696/volumes" Oct 31 20:01:22.288964 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.288851 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" podUID="f0c12fb7-e4fd-40f1-8a3d-6909084f0394" containerName="payload-processing" containerID="cri-o://9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe" gracePeriod=25 Oct 31 20:01:22.402327 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.402244 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" podUID="4b991a69-1862-41cf-b122-6bb782044784" containerName="payload-pre-processing" containerID="cri-o://8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242" gracePeriod=25 Oct 31 20:01:22.590566 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.587640 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" Oct 31 20:01:22.716911 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.716844 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" Oct 31 20:01:22.754905 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.752292 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-kube-api-access-vgdzk\" (UniqueName: \"kubernetes.io/projected/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-kube-api-access-vgdzk\") pod \"f0c12fb7-e4fd-40f1-8a3d-6909084f0394\" (UID: \"f0c12fb7-e4fd-40f1-8a3d-6909084f0394\") " Oct 31 20:01:22.754905 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.752454 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-plugins-config-volume\") pod \"f0c12fb7-e4fd-40f1-8a3d-6909084f0394\" (UID: \"f0c12fb7-e4fd-40f1-8a3d-6909084f0394\") " Oct 31 20:01:22.754905 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.753446 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-plugins-config-volume" pod "f0c12fb7-e4fd-40f1-8a3d-6909084f0394" (UID: "f0c12fb7-e4fd-40f1-8a3d-6909084f0394"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:22.755828 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.755799 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-kube-api-access-vgdzk" pod "f0c12fb7-e4fd-40f1-8a3d-6909084f0394" (UID: "f0c12fb7-e4fd-40f1-8a3d-6909084f0394"). InnerVolumeSpecName "kube-api-access-vgdzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:22.853816 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.853575 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b991a69-1862-41cf-b122-6bb782044784-kube-api-access-6949x\" (UniqueName: \"kubernetes.io/projected/4b991a69-1862-41cf-b122-6bb782044784-kube-api-access-6949x\") pod \"4b991a69-1862-41cf-b122-6bb782044784\" (UID: \"4b991a69-1862-41cf-b122-6bb782044784\") " Oct 31 20:01:22.853816 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.853695 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b991a69-1862-41cf-b122-6bb782044784-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b991a69-1862-41cf-b122-6bb782044784-plugins-config-volume\") pod \"4b991a69-1862-41cf-b122-6bb782044784\" (UID: \"4b991a69-1862-41cf-b122-6bb782044784\") " Oct 31 20:01:22.854288 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.854183 6107 generic.go:356] "Generic (PLEG): container finished" podID="f0c12fb7-e4fd-40f1-8a3d-6909084f0394" containerID="9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe" exitCode=0 Oct 31 20:01:22.854288 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.854240 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" Oct 31 20:01:22.854450 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.854296 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgdzk\" (UniqueName: \"kubernetes.io/projected/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-kube-api-access-vgdzk\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:22.854450 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.854316 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0c12fb7-e4fd-40f1-8a3d-6909084f0394-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:22.854450 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.854339 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" event={"ID":"f0c12fb7-e4fd-40f1-8a3d-6909084f0394","Type":"ContainerDied","Data":"9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe"} Oct 31 20:01:22.854450 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.854388 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk" event={"ID":"f0c12fb7-e4fd-40f1-8a3d-6909084f0394","Type":"ContainerDied","Data":"b724c156c701f6e470bd342c365a9276a0a2f122f1b3ea10e98b2dce91f2f017"} Oct 31 20:01:22.854450 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.854409 6107 scope.go:122] "RemoveContainer" containerID="9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe" Oct 31 20:01:22.855052 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.855025 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b991a69-1862-41cf-b122-6bb782044784-plugins-config-volume" pod "4b991a69-1862-41cf-b122-6bb782044784" (UID: "4b991a69-1862-41cf-b122-6bb782044784"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:22.856678 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.856654 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b991a69-1862-41cf-b122-6bb782044784-kube-api-access-6949x" pod "4b991a69-1862-41cf-b122-6bb782044784" (UID: "4b991a69-1862-41cf-b122-6bb782044784"). InnerVolumeSpecName "kube-api-access-6949x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:22.867120 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.867090 6107 generic.go:356] "Generic (PLEG): container finished" podID="4b991a69-1862-41cf-b122-6bb782044784" containerID="8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242" exitCode=0 Oct 31 20:01:22.867232 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.867209 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" Oct 31 20:01:22.867297 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.867232 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" event={"ID":"4b991a69-1862-41cf-b122-6bb782044784","Type":"ContainerDied","Data":"8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242"} Oct 31 20:01:22.867343 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.867301 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" event={"ID":"4b991a69-1862-41cf-b122-6bb782044784","Type":"ContainerDied","Data":"17ffd6d0f61cf124a4d13f463882122ebe60337712c840bfd00b030c91b78a19"} Oct 31 20:01:22.876569 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.876551 6107 scope.go:122] "RemoveContainer" containerID="9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe" Oct 31 20:01:22.876846 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:01:22.876823 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe\": container with ID starting with 9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe not found: ID does not exist" containerID="9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe" Oct 31 20:01:22.876974 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.876855 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe"} err="failed to get container status \"9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe\": rpc error: code = NotFound desc = could not find container \"9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe\": container with ID starting with 9265343dc10b6a27136a1c83b878be59e0fe2bc5cf3e6c6c5d74e80aed85dffe not found: ID does not exist" Oct 31 20:01:22.876974 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.876871 6107 scope.go:122] "RemoveContainer" containerID="8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242" Oct 31 20:01:22.886898 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.886850 6107 scope.go:122] "RemoveContainer" containerID="8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242" Oct 31 20:01:22.887211 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:01:22.887187 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242\": container with ID starting with 8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242 not found: ID does not exist" containerID="8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242" Oct 31 20:01:22.887287 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.887221 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242"} err="failed to get container status \"8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242\": rpc error: code = NotFound desc = could not find container \"8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242\": container with ID starting with 8ebb008ad5d2453cef8488a8094ac7598f36048cb4df3fb92c4a10cd3bec7242 not found: ID does not exist" Oct 31 20:01:22.956012 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.955978 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6949x\" (UniqueName: \"kubernetes.io/projected/4b991a69-1862-41cf-b122-6bb782044784-kube-api-access-6949x\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:22.956012 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:22.956004 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b991a69-1862-41cf-b122-6bb782044784-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:23.008064 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:23.007995 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm"] Oct 31 20:01:23.011349 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:23.011317 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm"] Oct 31 20:01:23.109739 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:23.109628 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk"] Oct 31 20:01:23.113314 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:23.113289 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dfe658-65bbf65875-6xvxk"] Oct 31 20:01:23.605937 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:23.605858 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dfe658-bf75889bc-g2kpm" podUID="4b991a69-1862-41cf-b122-6bb782044784" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.109:50052\" within 1s: context deadline exceeded" Oct 31 20:01:23.606400 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:01:23.605986 6107 logging.go:55] [core] [Channel #5075 SubChannel #5076]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.109:50052", ServerName: "10.133.0.109:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.109:50052: operation was canceled" Oct 31 20:01:24.490018 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:24.489983 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b991a69-1862-41cf-b122-6bb782044784" path="/var/lib/kubelet/pods/4b991a69-1862-41cf-b122-6bb782044784/volumes" Oct 31 20:01:24.490513 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:24.490491 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0c12fb7-e4fd-40f1-8a3d-6909084f0394" path="/var/lib/kubelet/pods/f0c12fb7-e4fd-40f1-8a3d-6909084f0394/volumes" Oct 31 20:01:36.347430 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:36.347387 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6"] Oct 31 20:01:36.459750 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:36.459713 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6"] Oct 31 20:01:41.348330 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.348255 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" podUID="b851ea3a-e54b-4119-b2d7-5452084005f2" containerName="payload-processing" containerID="cri-o://01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444" gracePeriod=25 Oct 31 20:01:41.460506 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.460466 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" podUID="d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62" containerName="payload-pre-processing" containerID="cri-o://88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d" gracePeriod=25 Oct 31 20:01:41.644217 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.644189 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" Oct 31 20:01:41.669178 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.669133 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b851ea3a-e54b-4119-b2d7-5452084005f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b851ea3a-e54b-4119-b2d7-5452084005f2-plugins-config-volume\") pod \"b851ea3a-e54b-4119-b2d7-5452084005f2\" (UID: \"b851ea3a-e54b-4119-b2d7-5452084005f2\") " Oct 31 20:01:41.669363 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.669348 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b851ea3a-e54b-4119-b2d7-5452084005f2-kube-api-access-tjrpv\" (UniqueName: \"kubernetes.io/projected/b851ea3a-e54b-4119-b2d7-5452084005f2-kube-api-access-tjrpv\") pod \"b851ea3a-e54b-4119-b2d7-5452084005f2\" (UID: \"b851ea3a-e54b-4119-b2d7-5452084005f2\") " Oct 31 20:01:41.670631 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.670602 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b851ea3a-e54b-4119-b2d7-5452084005f2-plugins-config-volume" pod "b851ea3a-e54b-4119-b2d7-5452084005f2" (UID: "b851ea3a-e54b-4119-b2d7-5452084005f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:41.672548 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.672491 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b851ea3a-e54b-4119-b2d7-5452084005f2-kube-api-access-tjrpv" pod "b851ea3a-e54b-4119-b2d7-5452084005f2" (UID: "b851ea3a-e54b-4119-b2d7-5452084005f2"). InnerVolumeSpecName "kube-api-access-tjrpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:41.752357 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.752328 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" Oct 31 20:01:41.770821 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.770781 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-plugins-config-volume\") pod \"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62\" (UID: \"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62\") " Oct 31 20:01:41.771044 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.770990 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-kube-api-access-smnx6\" (UniqueName: \"kubernetes.io/projected/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-kube-api-access-smnx6\") pod \"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62\" (UID: \"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62\") " Oct 31 20:01:41.771238 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.771214 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-plugins-config-volume" pod "d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62" (UID: "d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:41.771522 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.771505 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjrpv\" (UniqueName: \"kubernetes.io/projected/b851ea3a-e54b-4119-b2d7-5452084005f2-kube-api-access-tjrpv\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:41.771589 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.771530 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b851ea3a-e54b-4119-b2d7-5452084005f2-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:41.771589 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.771545 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:41.773458 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.773428 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-kube-api-access-smnx6" pod "d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62" (UID: "d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62"). InnerVolumeSpecName "kube-api-access-smnx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:41.872982 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:41.872870 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smnx6\" (UniqueName: \"kubernetes.io/projected/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62-kube-api-access-smnx6\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:42.024582 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.024539 6107 generic.go:356] "Generic (PLEG): container finished" podID="d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62" containerID="88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d" exitCode=0 Oct 31 20:01:42.024790 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.024601 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" Oct 31 20:01:42.024790 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.024626 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" event={"ID":"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62","Type":"ContainerDied","Data":"88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d"} Oct 31 20:01:42.024790 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.024686 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6" event={"ID":"d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62","Type":"ContainerDied","Data":"c75ceab1743a1f29ccb341024e2abc54be142def26b98da8efb9178d74ef2f4f"} Oct 31 20:01:42.024790 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.024714 6107 scope.go:122] "RemoveContainer" containerID="88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d" Oct 31 20:01:42.026708 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.026674 6107 generic.go:356] "Generic (PLEG): container finished" podID="b851ea3a-e54b-4119-b2d7-5452084005f2" containerID="01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444" exitCode=0 Oct 31 20:01:42.026851 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.026727 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" Oct 31 20:01:42.026851 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.026736 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" event={"ID":"b851ea3a-e54b-4119-b2d7-5452084005f2","Type":"ContainerDied","Data":"01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444"} Oct 31 20:01:42.026851 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.026787 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6" event={"ID":"b851ea3a-e54b-4119-b2d7-5452084005f2","Type":"ContainerDied","Data":"a526fb87183e07de5dd31b75f41fc550080e8ec8d5a155490bfc8b43b60c0b3c"} Oct 31 20:01:42.038312 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.038290 6107 scope.go:122] "RemoveContainer" containerID="88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d" Oct 31 20:01:42.038688 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:01:42.038660 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d\": container with ID starting with 88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d not found: ID does not exist" containerID="88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d" Oct 31 20:01:42.038816 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.038698 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d"} err="failed to get container status \"88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d\": rpc error: code = NotFound desc = could not find container \"88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d\": container with ID starting with 88a35d668ccd1a5e6f10b2d30a002855b1a159b421670ac34582cd06efefa63d not found: ID does not exist" Oct 31 20:01:42.038816 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.038724 6107 scope.go:122] "RemoveContainer" containerID="01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444" Oct 31 20:01:42.051293 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.051275 6107 scope.go:122] "RemoveContainer" containerID="01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444" Oct 31 20:01:42.051616 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:01:42.051597 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444\": container with ID starting with 01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444 not found: ID does not exist" containerID="01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444" Oct 31 20:01:42.051666 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.051626 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444"} err="failed to get container status \"01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444\": rpc error: code = NotFound desc = could not find container \"01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444\": container with ID starting with 01adbfc1fb595b5e0d57c6091afc1f5c217f3a730acc6ec78cadb3e14590c444 not found: ID does not exist" Oct 31 20:01:42.177201 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.177113 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6"] Oct 31 20:01:42.182468 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.182437 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5370f5-6b7b6bbf55-qzhd6"] Oct 31 20:01:42.283161 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.283124 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6"] Oct 31 20:01:42.286502 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.286469 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5370f5-77d9d9b86c-b7mb6"] Oct 31 20:01:42.498916 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.489953 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b851ea3a-e54b-4119-b2d7-5452084005f2" path="/var/lib/kubelet/pods/b851ea3a-e54b-4119-b2d7-5452084005f2/volumes" Oct 31 20:01:42.498916 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.490752 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62" path="/var/lib/kubelet/pods/d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62/volumes" Oct 31 20:01:42.655126 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.655090 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4"] Oct 31 20:01:42.759696 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:42.759605 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx"] Oct 31 20:01:47.655641 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:47.655598 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" podUID="0ec282e8-0fc4-4bdb-9212-9ab568c6200f" containerName="payload-processing" containerID="cri-o://30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4" gracePeriod=25 Oct 31 20:01:47.760241 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:47.760167 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" podUID="fda177cc-c3ca-42a4-9104-6da967a51a4f" containerName="payload-pre-processing" containerID="cri-o://cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11" gracePeriod=25 Oct 31 20:01:47.895917 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:01:47.894098 6107 logging.go:55] [core] [Channel #5197 SubChannel #5200]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.102:50052", ServerName: "10.133.0.102:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.102:50052: connect: connection refused" Oct 31 20:01:47.946146 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:47.946113 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" Oct 31 20:01:47.977952 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:47.977917 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-plugins-config-volume\") pod \"0ec282e8-0fc4-4bdb-9212-9ab568c6200f\" (UID: \"0ec282e8-0fc4-4bdb-9212-9ab568c6200f\") " Oct 31 20:01:47.978171 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:47.978136 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-kube-api-access-cr9lt\" (UniqueName: \"kubernetes.io/projected/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-kube-api-access-cr9lt\") pod \"0ec282e8-0fc4-4bdb-9212-9ab568c6200f\" (UID: \"0ec282e8-0fc4-4bdb-9212-9ab568c6200f\") " Oct 31 20:01:47.979531 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:47.979501 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-plugins-config-volume" pod "0ec282e8-0fc4-4bdb-9212-9ab568c6200f" (UID: "0ec282e8-0fc4-4bdb-9212-9ab568c6200f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:47.983150 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:47.983122 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-kube-api-access-cr9lt" pod "0ec282e8-0fc4-4bdb-9212-9ab568c6200f" (UID: "0ec282e8-0fc4-4bdb-9212-9ab568c6200f"). InnerVolumeSpecName "kube-api-access-cr9lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:48.059869 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.059841 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" Oct 31 20:01:48.079840 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.079807 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda177cc-c3ca-42a4-9104-6da967a51a4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda177cc-c3ca-42a4-9104-6da967a51a4f-plugins-config-volume\") pod \"fda177cc-c3ca-42a4-9104-6da967a51a4f\" (UID: \"fda177cc-c3ca-42a4-9104-6da967a51a4f\") " Oct 31 20:01:48.080066 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.079868 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda177cc-c3ca-42a4-9104-6da967a51a4f-kube-api-access-9cv8q\" (UniqueName: \"kubernetes.io/projected/fda177cc-c3ca-42a4-9104-6da967a51a4f-kube-api-access-9cv8q\") pod \"fda177cc-c3ca-42a4-9104-6da967a51a4f\" (UID: \"fda177cc-c3ca-42a4-9104-6da967a51a4f\") " Oct 31 20:01:48.080320 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.080291 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:48.080320 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.080310 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr9lt\" (UniqueName: \"kubernetes.io/projected/0ec282e8-0fc4-4bdb-9212-9ab568c6200f-kube-api-access-cr9lt\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:48.080320 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.080287 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda177cc-c3ca-42a4-9104-6da967a51a4f-plugins-config-volume" pod "fda177cc-c3ca-42a4-9104-6da967a51a4f" (UID: "fda177cc-c3ca-42a4-9104-6da967a51a4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:48.081594 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.081565 6107 generic.go:356] "Generic (PLEG): container finished" podID="0ec282e8-0fc4-4bdb-9212-9ab568c6200f" containerID="30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4" exitCode=0 Oct 31 20:01:48.081740 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.081686 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" Oct 31 20:01:48.081740 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.081700 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" event={"ID":"0ec282e8-0fc4-4bdb-9212-9ab568c6200f","Type":"ContainerDied","Data":"30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4"} Oct 31 20:01:48.081858 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.081761 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" event={"ID":"0ec282e8-0fc4-4bdb-9212-9ab568c6200f","Type":"ContainerDied","Data":"64abebc0c83316126045276816f47aea3b84654e0db3972d939ebc2872da0445"} Oct 31 20:01:48.081858 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.081793 6107 scope.go:122] "RemoveContainer" containerID="30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4" Oct 31 20:01:48.082558 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.082530 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda177cc-c3ca-42a4-9104-6da967a51a4f-kube-api-access-9cv8q" pod "fda177cc-c3ca-42a4-9104-6da967a51a4f" (UID: "fda177cc-c3ca-42a4-9104-6da967a51a4f"). InnerVolumeSpecName "kube-api-access-9cv8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:48.084348 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.084322 6107 generic.go:356] "Generic (PLEG): container finished" podID="fda177cc-c3ca-42a4-9104-6da967a51a4f" containerID="cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11" exitCode=0 Oct 31 20:01:48.084461 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.084365 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" Oct 31 20:01:48.084461 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.084413 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" event={"ID":"fda177cc-c3ca-42a4-9104-6da967a51a4f","Type":"ContainerDied","Data":"cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11"} Oct 31 20:01:48.084461 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.084455 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" event={"ID":"fda177cc-c3ca-42a4-9104-6da967a51a4f","Type":"ContainerDied","Data":"738211893a216e96299cc50631d3f460488253a212fcf41162ba94f59ff5198e"} Oct 31 20:01:48.095307 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.094578 6107 scope.go:122] "RemoveContainer" containerID="30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4" Oct 31 20:01:48.095307 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:01:48.094842 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4\": container with ID starting with 30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4 not found: ID does not exist" containerID="30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4" Oct 31 20:01:48.095307 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.094982 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4"} err="failed to get container status \"30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4\": rpc error: code = NotFound desc = could not find container \"30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4\": container with ID starting with 30f055ccbb712aa1e01c18cbb163eaa7ee74907964ba8f539fdaca304f34e6b4 not found: ID does not exist" Oct 31 20:01:48.095307 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.095011 6107 scope.go:122] "RemoveContainer" containerID="cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11" Oct 31 20:01:48.108308 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.108220 6107 scope.go:122] "RemoveContainer" containerID="cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11" Oct 31 20:01:48.108635 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:01:48.108560 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11\": container with ID starting with cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11 not found: ID does not exist" containerID="cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11" Oct 31 20:01:48.108635 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.108597 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11"} err="failed to get container status \"cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11\": rpc error: code = NotFound desc = could not find container \"cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11\": container with ID starting with cdca6aff414795ae5f24133729a2b912ba415e4e2ce1e3955b6c417b60fa3c11 not found: ID does not exist" Oct 31 20:01:48.181365 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.181248 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda177cc-c3ca-42a4-9104-6da967a51a4f-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:48.181365 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.181320 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cv8q\" (UniqueName: \"kubernetes.io/projected/fda177cc-c3ca-42a4-9104-6da967a51a4f-kube-api-access-9cv8q\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:48.269499 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.269463 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx"] Oct 31 20:01:48.274093 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.274016 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx"] Oct 31 20:01:48.378655 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.378618 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4"] Oct 31 20:01:48.381914 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.381876 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4"] Oct 31 20:01:48.487918 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.487871 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec282e8-0fc4-4bdb-9212-9ab568c6200f" path="/var/lib/kubelet/pods/0ec282e8-0fc4-4bdb-9212-9ab568c6200f/volumes" Oct 31 20:01:48.488323 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.488310 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda177cc-c3ca-42a4-9104-6da967a51a4f" path="/var/lib/kubelet/pods/fda177cc-c3ca-42a4-9104-6da967a51a4f/volumes" Oct 31 20:01:48.894288 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.894206 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-ddb5a8-5c45c744ff-rw5wx" podUID="fda177cc-c3ca-42a4-9104-6da967a51a4f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.102:50052\" within 1s: context deadline exceeded" Oct 31 20:01:48.894665 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:01:48.894215 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ddb5a8-67db976fb7-ldqp4" podUID="0ec282e8-0fc4-4bdb-9212-9ab568c6200f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.101:50052\" within 1s: context deadline exceeded" Oct 31 20:01:48.894665 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:01:48.894308 6107 logging.go:55] [core] [Channel #5198 SubChannel #5199]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.101:50052", ServerName: "10.133.0.101:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.101:50052: operation was canceled" Oct 31 20:01:50.406418 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:01:50.406373 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:02:00.727410 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.727371 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc"] Oct 31 20:02:00.728106 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728084 6107 cpu_manager.go:414] "containerMap: removing container" podUID="81d92768-7aee-4fa2-9777-d13bcc073772" containerName="payload-pre-processing" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728110 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d92768-7aee-4fa2-9777-d13bcc073772" containerName="payload-pre-processing" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728137 6107 cpu_manager.go:414] "containerMap: removing container" podUID="049cf604-6e9d-435d-8f6f-e141c1fb289b" containerName="maas-api" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728145 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049cf604-6e9d-435d-8f6f-e141c1fb289b" containerName="maas-api" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728154 6107 cpu_manager.go:414] "containerMap: removing container" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728160 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728170 6107 cpu_manager.go:414] "containerMap: removing container" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728177 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728187 6107 cpu_manager.go:414] "containerMap: removing container" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728191 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728200 6107 cpu_manager.go:414] "containerMap: removing container" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728204 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728212 6107 cpu_manager.go:414] "containerMap: removing container" podUID="0ec282e8-0fc4-4bdb-9212-9ab568c6200f" containerName="payload-processing" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728216 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec282e8-0fc4-4bdb-9212-9ab568c6200f" containerName="payload-processing" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728224 6107 cpu_manager.go:414] "containerMap: removing container" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728228 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728234 6107 cpu_manager.go:414] "containerMap: removing container" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728239 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728245 6107 cpu_manager.go:414] "containerMap: removing container" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" containerName="cleanup" Oct 31 20:02:00.728243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728249 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728255 6107 cpu_manager.go:414] "containerMap: removing container" podUID="f0c12fb7-e4fd-40f1-8a3d-6909084f0394" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728262 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c12fb7-e4fd-40f1-8a3d-6909084f0394" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728270 6107 cpu_manager.go:414] "containerMap: removing container" podUID="fda177cc-c3ca-42a4-9104-6da967a51a4f" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728275 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda177cc-c3ca-42a4-9104-6da967a51a4f" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728284 6107 cpu_manager.go:414] "containerMap: removing container" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728288 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728296 6107 cpu_manager.go:414] "containerMap: removing container" podUID="c17338a4-ac0b-4b76-a622-28fe74269fa9" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728300 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17338a4-ac0b-4b76-a622-28fe74269fa9" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728306 6107 cpu_manager.go:414] "containerMap: removing container" podUID="f7d03485-903b-480b-a923-08a9fd632e7b" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728310 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d03485-903b-480b-a923-08a9fd632e7b" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728316 6107 cpu_manager.go:414] "containerMap: removing container" podUID="4b991a69-1862-41cf-b122-6bb782044784" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728320 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b991a69-1862-41cf-b122-6bb782044784" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728327 6107 cpu_manager.go:414] "containerMap: removing container" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728331 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728342 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d68b1ca1-2e79-4786-9799-94151203f153" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728346 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68b1ca1-2e79-4786-9799-94151203f153" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728354 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d68b1ca1-2e79-4786-9799-94151203f153" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728358 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68b1ca1-2e79-4786-9799-94151203f153" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728368 6107 cpu_manager.go:414] "containerMap: removing container" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728372 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728385 6107 cpu_manager.go:414] "containerMap: removing container" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728389 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728397 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728401 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728409 6107 cpu_manager.go:414] "containerMap: removing container" podUID="63d0bee6-b082-4ab9-af16-d903398f9696" containerName="maas-api" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728413 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d0bee6-b082-4ab9-af16-d903398f9696" containerName="maas-api" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728421 6107 cpu_manager.go:414] "containerMap: removing container" podUID="b851ea3a-e54b-4119-b2d7-5452084005f2" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728425 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b851ea3a-e54b-4119-b2d7-5452084005f2" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728431 6107 cpu_manager.go:414] "containerMap: removing container" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728434 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728442 6107 cpu_manager.go:414] "containerMap: removing container" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728445 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728453 6107 cpu_manager.go:414] "containerMap: removing container" podUID="75491f8d-9f81-4f0b-afbf-d2a63b7b6731" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728458 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75491f8d-9f81-4f0b-afbf-d2a63b7b6731" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728552 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728560 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728566 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec282e8-0fc4-4bdb-9212-9ab568c6200f" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728574 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728578 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728583 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728588 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728593 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c12fb7-e4fd-40f1-8a3d-6909084f0394" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728599 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728604 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda177cc-c3ca-42a4-9104-6da967a51a4f" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728610 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728616 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17338a4-ac0b-4b76-a622-28fe74269fa9" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728622 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d03485-903b-480b-a923-08a9fd632e7b" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728627 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b991a69-1862-41cf-b122-6bb782044784" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728634 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68b1ca1-2e79-4786-9799-94151203f153" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728640 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68b1ca1-2e79-4786-9799-94151203f153" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728646 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68b1ca1-2e79-4786-9799-94151203f153" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728651 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728656 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728662 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728668 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6428a1a-6ae9-4fb6-bb42-16e7dd39bf62" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728674 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d0bee6-b082-4ab9-af16-d903398f9696" containerName="maas-api" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728678 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="b851ea3a-e54b-4119-b2d7-5452084005f2" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728684 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728689 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="75491f8d-9f81-4f0b-afbf-d2a63b7b6731" containerName="payload-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728696 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d92768-7aee-4fa2-9777-d13bcc073772" containerName="payload-pre-processing" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728702 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="049cf604-6e9d-435d-8f6f-e141c1fb289b" containerName="maas-api" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728708 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" containerName="cleanup" Oct 31 20:02:00.728775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.728713 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" containerName="cleanup" Oct 31 20:02:00.733892 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.733861 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:00.787367 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.787325 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc"] Oct 31 20:02:00.824767 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.824736 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fd95d7-serving-cert\"" Oct 31 20:02:00.874329 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.874296 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ccf1df-3361-4057-980e-1b38a5fd5ea7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:00.874496 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.874378 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzw9k\" (UniqueName: \"kubernetes.io/projected/82ccf1df-3361-4057-980e-1b38a5fd5ea7-kube-api-access-qzw9k\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:00.874496 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.874484 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:00.874590 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.874530 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-metrics-certs\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:00.979578 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.976071 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ccf1df-3361-4057-980e-1b38a5fd5ea7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:00.979578 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.976180 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzw9k\" (UniqueName: \"kubernetes.io/projected/82ccf1df-3361-4057-980e-1b38a5fd5ea7-kube-api-access-qzw9k\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:00.979578 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.976264 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:00.979578 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.976329 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-metrics-certs\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:00.981983 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:02:00.981935 6107 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-serving-cert: secret "maas-api-e2e-inf-a-fd95d7-serving-cert" not found Oct 31 20:02:00.983523 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:02:00.983500 6107 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-maas-api-tls podName:82ccf1df-3361-4057-980e-1b38a5fd5ea7 nodeName:}" failed. No retries permitted until 2026-10-31 20:02:01.483463069 +0000 UTC m=+2693.116856963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-maas-api-tls") pod "maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" (UID: "82ccf1df-3361-4057-980e-1b38a5fd5ea7") : secret "maas-api-e2e-inf-a-fd95d7-serving-cert" not found Oct 31 20:02:00.983797 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.983764 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ccf1df-3361-4057-980e-1b38a5fd5ea7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:00.985628 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:00.985595 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-metrics-certs\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:01.045291 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:01.045248 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzw9k\" (UniqueName: \"kubernetes.io/projected/82ccf1df-3361-4057-980e-1b38a5fd5ea7-kube-api-access-qzw9k\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:01.483600 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:01.483567 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:01.486045 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:01.486019 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:01.649806 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:01.649764 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:02.027304 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:02.027270 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc"] Oct 31 20:02:02.028475 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:02:02.028440 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ccf1df_3361_4057_980e_1b38a5fd5ea7.slice/crio-bd52c59b2ca8522a64de8a31d7bd8280ea8338271a6f199828e68291691c487f WatchSource:0}: Error finding container bd52c59b2ca8522a64de8a31d7bd8280ea8338271a6f199828e68291691c487f: Status 404 returned error can't find the container with id bd52c59b2ca8522a64de8a31d7bd8280ea8338271a6f199828e68291691c487f Oct 31 20:02:02.195956 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:02.195911 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" event={"ID":"82ccf1df-3361-4057-980e-1b38a5fd5ea7","Type":"ContainerStarted","Data":"bd52c59b2ca8522a64de8a31d7bd8280ea8338271a6f199828e68291691c487f"} Oct 31 20:02:03.205505 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:03.205402 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" event={"ID":"82ccf1df-3361-4057-980e-1b38a5fd5ea7","Type":"ContainerStarted","Data":"859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2"} Oct 31 20:02:03.205505 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:03.205467 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:03.312461 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:03.312406 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" podStartSLOduration=2.461482277 podStartE2EDuration="3.312392186s" totalImagesPullingTime="850.909909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:03.308910504 +0000 UTC m=+2694.942304409" watchObservedRunningTime="2026-10-31 20:02:03.312392186 +0000 UTC m=+2694.945786094" Oct 31 20:02:04.606356 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:04.606314 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7"] Oct 31 20:02:04.713543 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:04.713511 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz"] Oct 31 20:02:09.218712 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:09.218682 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:02:09.607864 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:09.607778 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" podUID="3b785406-50fa-48e4-aa62-d36c95495ca4" containerName="payload-processing" containerID="cri-o://a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3" gracePeriod=25 Oct 31 20:02:09.713938 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:09.713840 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" podUID="d9267cfb-f5bf-4b85-bfef-9327801c261b" containerName="payload-pre-processing" containerID="cri-o://d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32" gracePeriod=25 Oct 31 20:02:09.914055 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:09.908929 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" Oct 31 20:02:10.020728 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.020700 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" Oct 31 20:02:10.033179 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.033140 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b785406-50fa-48e4-aa62-d36c95495ca4-kube-api-access-6fnlw\" (UniqueName: \"kubernetes.io/projected/3b785406-50fa-48e4-aa62-d36c95495ca4-kube-api-access-6fnlw\") pod \"3b785406-50fa-48e4-aa62-d36c95495ca4\" (UID: \"3b785406-50fa-48e4-aa62-d36c95495ca4\") " Oct 31 20:02:10.033343 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.033190 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b785406-50fa-48e4-aa62-d36c95495ca4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b785406-50fa-48e4-aa62-d36c95495ca4-plugins-config-volume\") pod \"3b785406-50fa-48e4-aa62-d36c95495ca4\" (UID: \"3b785406-50fa-48e4-aa62-d36c95495ca4\") " Oct 31 20:02:10.033874 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.033848 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b785406-50fa-48e4-aa62-d36c95495ca4-plugins-config-volume" pod "3b785406-50fa-48e4-aa62-d36c95495ca4" (UID: "3b785406-50fa-48e4-aa62-d36c95495ca4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:10.036610 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.036583 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b785406-50fa-48e4-aa62-d36c95495ca4-kube-api-access-6fnlw" pod "3b785406-50fa-48e4-aa62-d36c95495ca4" (UID: "3b785406-50fa-48e4-aa62-d36c95495ca4"). InnerVolumeSpecName "kube-api-access-6fnlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:10.134362 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.134284 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9267cfb-f5bf-4b85-bfef-9327801c261b-kube-api-access-zc8bh\" (UniqueName: \"kubernetes.io/projected/d9267cfb-f5bf-4b85-bfef-9327801c261b-kube-api-access-zc8bh\") pod \"d9267cfb-f5bf-4b85-bfef-9327801c261b\" (UID: \"d9267cfb-f5bf-4b85-bfef-9327801c261b\") " Oct 31 20:02:10.134508 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.134409 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9267cfb-f5bf-4b85-bfef-9327801c261b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9267cfb-f5bf-4b85-bfef-9327801c261b-plugins-config-volume\") pod \"d9267cfb-f5bf-4b85-bfef-9327801c261b\" (UID: \"d9267cfb-f5bf-4b85-bfef-9327801c261b\") " Oct 31 20:02:10.134830 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.134799 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9267cfb-f5bf-4b85-bfef-9327801c261b-plugins-config-volume" pod "d9267cfb-f5bf-4b85-bfef-9327801c261b" (UID: "d9267cfb-f5bf-4b85-bfef-9327801c261b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:10.135002 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.134866 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fnlw\" (UniqueName: \"kubernetes.io/projected/3b785406-50fa-48e4-aa62-d36c95495ca4-kube-api-access-6fnlw\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:10.135002 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.134910 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b785406-50fa-48e4-aa62-d36c95495ca4-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:10.136866 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.136839 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9267cfb-f5bf-4b85-bfef-9327801c261b-kube-api-access-zc8bh" pod "d9267cfb-f5bf-4b85-bfef-9327801c261b" (UID: "d9267cfb-f5bf-4b85-bfef-9327801c261b"). InnerVolumeSpecName "kube-api-access-zc8bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:10.236534 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.236499 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9267cfb-f5bf-4b85-bfef-9327801c261b-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:10.236534 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.236529 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8bh\" (UniqueName: \"kubernetes.io/projected/d9267cfb-f5bf-4b85-bfef-9327801c261b-kube-api-access-zc8bh\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:10.259478 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.259439 6107 generic.go:356] "Generic (PLEG): container finished" podID="3b785406-50fa-48e4-aa62-d36c95495ca4" containerID="a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3" exitCode=0 Oct 31 20:02:10.259653 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.259494 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" Oct 31 20:02:10.259653 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.259520 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" event={"ID":"3b785406-50fa-48e4-aa62-d36c95495ca4","Type":"ContainerDied","Data":"a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3"} Oct 31 20:02:10.259653 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.259570 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7" event={"ID":"3b785406-50fa-48e4-aa62-d36c95495ca4","Type":"ContainerDied","Data":"58a155c904df8b1a386fbe5f846a861d611e4365ad71ca3f2cf41bb193cb63b3"} Oct 31 20:02:10.259653 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.259597 6107 scope.go:122] "RemoveContainer" containerID="a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3" Oct 31 20:02:10.261072 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.261047 6107 generic.go:356] "Generic (PLEG): container finished" podID="d9267cfb-f5bf-4b85-bfef-9327801c261b" containerID="d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32" exitCode=0 Oct 31 20:02:10.261212 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.261092 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" Oct 31 20:02:10.261212 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.261126 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" event={"ID":"d9267cfb-f5bf-4b85-bfef-9327801c261b","Type":"ContainerDied","Data":"d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32"} Oct 31 20:02:10.261212 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.261169 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz" event={"ID":"d9267cfb-f5bf-4b85-bfef-9327801c261b","Type":"ContainerDied","Data":"5b7b8f3d36e4a2c987d3a8127f8f78d897fde5b2eed0e056a476d51baf473837"} Oct 31 20:02:10.271741 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.271725 6107 scope.go:122] "RemoveContainer" containerID="a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3" Oct 31 20:02:10.272055 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:02:10.272035 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3\": container with ID starting with a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3 not found: ID does not exist" containerID="a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3" Oct 31 20:02:10.272118 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.272063 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3"} err="failed to get container status \"a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3\": rpc error: code = NotFound desc = could not find container \"a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3\": container with ID starting with a631bb3ef3bd2a788b931b92bd1f958e5135f2d1701c120d54958c5d7abf92d3 not found: ID does not exist" Oct 31 20:02:10.272118 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.272081 6107 scope.go:122] "RemoveContainer" containerID="d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32" Oct 31 20:02:10.282827 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.282811 6107 scope.go:122] "RemoveContainer" containerID="d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32" Oct 31 20:02:10.283182 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:02:10.283164 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32\": container with ID starting with d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32 not found: ID does not exist" containerID="d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32" Oct 31 20:02:10.283256 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.283189 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32"} err="failed to get container status \"d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32\": rpc error: code = NotFound desc = could not find container \"d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32\": container with ID starting with d55f154f8e3acbd57c5ca30357096dc7a7b7f73f9d7955f4e0b176a291fb4c32 not found: ID does not exist" Oct 31 20:02:10.410762 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.410676 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7"] Oct 31 20:02:10.413763 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.413731 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-650cf3-8476cb4dbc-lhjt7"] Oct 31 20:02:10.488382 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.488350 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b785406-50fa-48e4-aa62-d36c95495ca4" path="/var/lib/kubelet/pods/3b785406-50fa-48e4-aa62-d36c95495ca4/volumes" Oct 31 20:02:10.512753 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.512719 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz"] Oct 31 20:02:10.515849 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:10.515823 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-650cf3-5f974ff49f-n7xqz"] Oct 31 20:02:12.487992 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:12.487964 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9267cfb-f5bf-4b85-bfef-9327801c261b" path="/var/lib/kubelet/pods/d9267cfb-f5bf-4b85-bfef-9327801c261b/volumes" Oct 31 20:02:20.410743 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:02:20.410541 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:02:29.417086 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:29.417052 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr"] Oct 31 20:02:31.860714 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.860676 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf"] Oct 31 20:02:31.861424 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861406 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d68b1ca1-2e79-4786-9799-94151203f153" containerName="cleanup" Oct 31 20:02:31.861424 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861425 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68b1ca1-2e79-4786-9799-94151203f153" containerName="cleanup" Oct 31 20:02:31.861544 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861443 6107 cpu_manager.go:414] "containerMap: removing container" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" containerName="cleanup" Oct 31 20:02:31.861544 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861452 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" containerName="cleanup" Oct 31 20:02:31.861544 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861473 6107 cpu_manager.go:414] "containerMap: removing container" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" containerName="cleanup" Oct 31 20:02:31.861544 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861479 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ed662e-cdac-4394-ac95-4b103b48bbff" containerName="cleanup" Oct 31 20:02:31.861544 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861491 6107 cpu_manager.go:414] "containerMap: removing container" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" containerName="cleanup" Oct 31 20:02:31.861544 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861496 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" containerName="cleanup" Oct 31 20:02:31.861544 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861535 6107 cpu_manager.go:414] "containerMap: removing container" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" containerName="cleanup" Oct 31 20:02:31.861544 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861542 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" containerName="cleanup" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861557 6107 cpu_manager.go:414] "containerMap: removing container" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" containerName="cleanup" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861562 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" containerName="cleanup" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861573 6107 cpu_manager.go:414] "containerMap: removing container" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" containerName="cleanup" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861578 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" containerName="cleanup" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861588 6107 cpu_manager.go:414] "containerMap: removing container" podUID="3b785406-50fa-48e4-aa62-d36c95495ca4" containerName="payload-processing" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861592 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b785406-50fa-48e4-aa62-d36c95495ca4" containerName="payload-processing" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861599 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d9267cfb-f5bf-4b85-bfef-9327801c261b" containerName="payload-pre-processing" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861603 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9267cfb-f5bf-4b85-bfef-9327801c261b" containerName="payload-pre-processing" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861611 6107 cpu_manager.go:414] "containerMap: removing container" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861615 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861735 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="29dfb5f8-7c03-427a-aa3c-4181855ed5fa" containerName="cleanup" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861745 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="620ec382-3553-4b53-8e0e-355469f9b0e5" containerName="cleanup" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861753 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3a32c2-060e-4568-9744-1dd49c0b4cba" containerName="cleanup" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861759 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b785406-50fa-48e4-aa62-d36c95495ca4" containerName="payload-processing" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861765 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9267cfb-f5bf-4b85-bfef-9327801c261b" containerName="payload-pre-processing" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861771 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="63dee691-27c6-4a5f-882c-fc3ed46cbeb1" containerName="istio-proxy" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861780 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c26d2d-3fa6-403e-9aa1-611dff153b14" containerName="cleanup" Oct 31 20:02:31.861817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.861790 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0e177d-af3a-41e8-bcd8-e963493aa6ca" containerName="cleanup" Oct 31 20:02:31.867034 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.867015 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:31.915180 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.915143 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fd95d7-openshift-default-dockercfg-cvnc8\"" Oct 31 20:02:31.966920 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.966859 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf"] Oct 31 20:02:31.971258 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.971230 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-istio-token\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:31.971390 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.971272 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-certs\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:31.971390 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.971372 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-credential-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:31.971524 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.971480 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5921ad77-6716-48a1-aac2-3c9a83497866-istiod-ca-cert\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:31.971602 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.971579 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-envoy\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:31.971702 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.971684 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlx4s\" (UniqueName: \"kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-kube-api-access-xlx4s\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:31.971765 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.971750 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:31.971818 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.971787 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-data\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:31.971948 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:31.971863 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5921ad77-6716-48a1-aac2-3c9a83497866-istio-podinfo\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.073453 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.073417 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-certs\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.073671 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.073478 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-credential-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.073671 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.073524 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5921ad77-6716-48a1-aac2-3c9a83497866-istiod-ca-cert\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.073671 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.073579 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-envoy\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.073671 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.073663 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlx4s\" (UniqueName: \"kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-kube-api-access-xlx4s\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.073978 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.073740 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.073978 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.073788 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-data\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.073978 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.073828 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5921ad77-6716-48a1-aac2-3c9a83497866-istio-podinfo\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.073978 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.073874 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-certs\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.074177 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.074012 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-istio-token\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.074233 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.074177 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-data\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.074233 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.074209 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.074358 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.074323 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-credential-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.074358 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.074345 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5921ad77-6716-48a1-aac2-3c9a83497866-istiod-ca-cert\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.076186 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.076165 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-envoy\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.076515 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.076496 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5921ad77-6716-48a1-aac2-3c9a83497866-istio-podinfo\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.127624 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.127551 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-istio-token\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.127777 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.127660 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlx4s\" (UniqueName: \"kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-kube-api-access-xlx4s\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.181778 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.181745 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:32.608304 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.608266 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf"] Oct 31 20:02:32.608751 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:02:32.608713 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921ad77_6716_48a1_aac2_3c9a83497866.slice/crio-9d1b77a7f891c1601246e031cd989fd2986161d753ca48499cb8ce58cc20d1eb WatchSource:0}: Error finding container 9d1b77a7f891c1601246e031cd989fd2986161d753ca48499cb8ce58cc20d1eb: Status 404 returned error can't find the container with id 9d1b77a7f891c1601246e031cd989fd2986161d753ca48499cb8ce58cc20d1eb Oct 31 20:02:32.611547 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.611514 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:02:32.611638 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.611579 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:02:32.611638 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:32.611609 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:02:33.442546 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:33.442503 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" event={"ID":"5921ad77-6716-48a1-aac2-3c9a83497866","Type":"ContainerStarted","Data":"0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99"} Oct 31 20:02:33.443005 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:33.442572 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" event={"ID":"5921ad77-6716-48a1-aac2-3c9a83497866","Type":"ContainerStarted","Data":"9d1b77a7f891c1601246e031cd989fd2986161d753ca48499cb8ce58cc20d1eb"} Oct 31 20:02:33.620904 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:33.618838 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" podStartSLOduration=2.618812931 podStartE2EDuration="2.618812931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:33.599341721 +0000 UTC m=+2725.232735627" watchObservedRunningTime="2026-10-31 20:02:33.618812931 +0000 UTC m=+2725.252206838" Oct 31 20:02:34.182683 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:34.182647 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:34.184644 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:34.184620 6107 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 20:02:34.184703 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:34.184666 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" podUID="5921ad77-6716-48a1-aac2-3c9a83497866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 20:02:34.422231 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:34.422178 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" podUID="f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8" containerName="payload-processing" containerID="cri-o://ef3b627586ea08bbf9f88fa20358d1b5a10bbc42133c3f5c964637b9abf6abad" gracePeriod=25 Oct 31 20:02:35.183045 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:35.183011 6107 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 20:02:35.183427 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:35.183064 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" podUID="5921ad77-6716-48a1-aac2-3c9a83497866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 20:02:36.182345 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:36.182308 6107 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 20:02:36.182530 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:36.182361 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" podUID="5921ad77-6716-48a1-aac2-3c9a83497866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 20:02:36.396521 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:36.396487 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" podUID="f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:02:37.182749 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:37.182711 6107 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 20:02:37.182966 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:37.182767 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" podUID="5921ad77-6716-48a1-aac2-3c9a83497866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 20:02:38.182899 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:38.182848 6107 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 20:02:38.183351 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:38.182932 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" podUID="5921ad77-6716-48a1-aac2-3c9a83497866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 20:02:39.188607 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:39.188572 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:39.189098 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:39.188934 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:39.190082 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:39.190058 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:02:40.630517 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:40.630486 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm"] Oct 31 20:02:40.631081 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:40.630726 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" containerID="cri-o://f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f" gracePeriod=30 Oct 31 20:02:45.911300 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.911238 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 20:02:45.995284 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995236 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-credential-socket\") pod \"a6e2b24f-63f6-40db-aca5-74376cb072f1\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " Oct 31 20:02:45.995490 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995344 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-socket\") pod \"a6e2b24f-63f6-40db-aca5-74376cb072f1\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " Oct 31 20:02:45.995490 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995403 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-envoy\") pod \"a6e2b24f-63f6-40db-aca5-74376cb072f1\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " Oct 31 20:02:45.995490 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995476 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e2b24f-63f6-40db-aca5-74376cb072f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e2b24f-63f6-40db-aca5-74376cb072f1-istiod-ca-cert\") pod \"a6e2b24f-63f6-40db-aca5-74376cb072f1\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " Oct 31 20:02:45.995658 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995527 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-kube-api-access-r5mqd\" (UniqueName: \"kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-kube-api-access-r5mqd\") pod \"a6e2b24f-63f6-40db-aca5-74376cb072f1\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " Oct 31 20:02:45.995658 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995543 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-credential-socket" pod "a6e2b24f-63f6-40db-aca5-74376cb072f1" (UID: "a6e2b24f-63f6-40db-aca5-74376cb072f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:45.995658 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995579 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-certs\") pod \"a6e2b24f-63f6-40db-aca5-74376cb072f1\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " Oct 31 20:02:45.995817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995662 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-podinfo\") pod \"a6e2b24f-63f6-40db-aca5-74376cb072f1\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " Oct 31 20:02:45.995817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995691 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-token\" (UniqueName: \"kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-token\") pod \"a6e2b24f-63f6-40db-aca5-74376cb072f1\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " Oct 31 20:02:45.995817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995671 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-socket" pod "a6e2b24f-63f6-40db-aca5-74376cb072f1" (UID: "a6e2b24f-63f6-40db-aca5-74376cb072f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:45.995817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995791 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e2b24f-63f6-40db-aca5-74376cb072f1-istiod-ca-cert" pod "a6e2b24f-63f6-40db-aca5-74376cb072f1" (UID: "a6e2b24f-63f6-40db-aca5-74376cb072f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:45.996047 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995856 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-data\") pod \"a6e2b24f-63f6-40db-aca5-74376cb072f1\" (UID: \"a6e2b24f-63f6-40db-aca5-74376cb072f1\") " Oct 31 20:02:45.996047 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.995970 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-certs" pod "a6e2b24f-63f6-40db-aca5-74376cb072f1" (UID: "a6e2b24f-63f6-40db-aca5-74376cb072f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:45.996176 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.996154 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-data" pod "a6e2b24f-63f6-40db-aca5-74376cb072f1" (UID: "a6e2b24f-63f6-40db-aca5-74376cb072f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:45.996537 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.996516 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:45.996645 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.996542 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:45.996645 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.996560 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:45.996645 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.996576 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e2b24f-63f6-40db-aca5-74376cb072f1-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:45.996645 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.996589 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:45.998077 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.998046 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-podinfo" pod "a6e2b24f-63f6-40db-aca5-74376cb072f1" (UID: "a6e2b24f-63f6-40db-aca5-74376cb072f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:02:45.998203 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.998182 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-envoy" pod "a6e2b24f-63f6-40db-aca5-74376cb072f1" (UID: "a6e2b24f-63f6-40db-aca5-74376cb072f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:45.999332 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.999310 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-kube-api-access-r5mqd" pod "a6e2b24f-63f6-40db-aca5-74376cb072f1" (UID: "a6e2b24f-63f6-40db-aca5-74376cb072f1"). InnerVolumeSpecName "kube-api-access-r5mqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:45.999446 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:45.999363 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-token" pod "a6e2b24f-63f6-40db-aca5-74376cb072f1" (UID: "a6e2b24f-63f6-40db-aca5-74376cb072f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:46.098424 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.098393 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:46.098424 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.098422 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5mqd\" (UniqueName: \"kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-kube-api-access-r5mqd\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:46.098424 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.098432 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:46.098671 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.098440 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e2b24f-63f6-40db-aca5-74376cb072f1-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:46.396895 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.396841 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" podUID="f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:02:46.535897 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.535854 6107 generic.go:356] "Generic (PLEG): container finished" podID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerID="f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f" exitCode=0 Oct 31 20:02:46.536072 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.535941 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" Oct 31 20:02:46.536144 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.536125 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" event={"ID":"a6e2b24f-63f6-40db-aca5-74376cb072f1","Type":"ContainerDied","Data":"f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f"} Oct 31 20:02:46.536201 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.536170 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm" event={"ID":"a6e2b24f-63f6-40db-aca5-74376cb072f1","Type":"ContainerDied","Data":"7c6eabcbcd231e35356a6ed2f7bb34cf60f6fc6132dfe29a714cd647405d0f9e"} Oct 31 20:02:46.536201 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.536197 6107 scope.go:122] "RemoveContainer" containerID="f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f" Oct 31 20:02:46.549238 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.549206 6107 scope.go:122] "RemoveContainer" containerID="f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f" Oct 31 20:02:46.549762 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:02:46.549731 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f\": container with ID starting with f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f not found: ID does not exist" containerID="f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f" Oct 31 20:02:46.550041 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.549774 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f"} err="failed to get container status \"f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f\": rpc error: code = NotFound desc = could not find container \"f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f\": container with ID starting with f88fa407bae57fa31162f46b9703fb5d7626c08e52eb8bb8f16303d4a48a702f not found: ID does not exist" Oct 31 20:02:46.700042 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.699948 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm"] Oct 31 20:02:46.712924 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:46.712873 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-bgtmm"] Oct 31 20:02:48.488463 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:48.488423 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" path="/var/lib/kubelet/pods/a6e2b24f-63f6-40db-aca5-74376cb072f1/volumes" Oct 31 20:02:50.409112 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:02:50.409062 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:02:54.599488 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:54.599457 6107 generic.go:356] "Generic (PLEG): container finished" podID="f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8" containerID="ef3b627586ea08bbf9f88fa20358d1b5a10bbc42133c3f5c964637b9abf6abad" exitCode=0 Oct 31 20:02:54.599851 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:54.599515 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" event={"ID":"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8","Type":"ContainerDied","Data":"ef3b627586ea08bbf9f88fa20358d1b5a10bbc42133c3f5c964637b9abf6abad"} Oct 31 20:02:54.699593 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:54.699533 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" Oct 31 20:02:54.815601 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:54.815566 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-plugins-config-volume\") pod \"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8\" (UID: \"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8\") " Oct 31 20:02:54.815761 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:54.815662 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-kube-api-access-5xzrr\" (UniqueName: \"kubernetes.io/projected/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-kube-api-access-5xzrr\") pod \"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8\" (UID: \"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8\") " Oct 31 20:02:54.815961 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:54.815939 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-plugins-config-volume" pod "f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8" (UID: "f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:54.816255 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:54.816237 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:54.817755 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:54.817727 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-kube-api-access-5xzrr" pod "f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8" (UID: "f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8"). InnerVolumeSpecName "kube-api-access-5xzrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:54.917144 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:54.917112 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xzrr\" (UniqueName: \"kubernetes.io/projected/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8-kube-api-access-5xzrr\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:55.614908 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:55.611500 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" Oct 31 20:02:55.614908 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:55.611527 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr" event={"ID":"f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8","Type":"ContainerDied","Data":"2577d6a9efbf4ddfe235fb8d467cfe33547feb707e3ff4851793cbc7b89a3e70"} Oct 31 20:02:55.614908 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:55.611590 6107 scope.go:122] "RemoveContainer" containerID="ef3b627586ea08bbf9f88fa20358d1b5a10bbc42133c3f5c964637b9abf6abad" Oct 31 20:02:55.785134 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:55.785030 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr"] Oct 31 20:02:55.816153 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:55.816120 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7cfafd-6c7fbc9c69-rnkbr"] Oct 31 20:02:56.490997 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:02:56.490954 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8" path="/var/lib/kubelet/pods/f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8/volumes" Oct 31 20:03:04.021824 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.021786 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96"] Oct 31 20:03:04.022328 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.022042 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" podUID="d6654dc0-fdbd-44b9-adfd-1dde68a708f9" containerName="maas-api" containerID="cri-o://f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c" gracePeriod=30 Oct 31 20:03:04.314871 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.314836 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 20:03:04.361022 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.360997 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-kube-api-access-rj8zc\" (UniqueName: \"kubernetes.io/projected/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-kube-api-access-rj8zc\") pod \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " Oct 31 20:03:04.361166 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.361131 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-metrics-certs\") pod \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " Oct 31 20:03:04.361212 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.361181 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-openshift-service-ca\") pod \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " Oct 31 20:03:04.361254 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.361229 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-maas-api-tls\") pod \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\" (UID: \"d6654dc0-fdbd-44b9-adfd-1dde68a708f9\") " Oct 31 20:03:04.361560 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.361535 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-openshift-service-ca" pod "d6654dc0-fdbd-44b9-adfd-1dde68a708f9" (UID: "d6654dc0-fdbd-44b9-adfd-1dde68a708f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:04.363201 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.363177 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-metrics-certs" pod "d6654dc0-fdbd-44b9-adfd-1dde68a708f9" (UID: "d6654dc0-fdbd-44b9-adfd-1dde68a708f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:03:04.363303 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.363284 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-kube-api-access-rj8zc" pod "d6654dc0-fdbd-44b9-adfd-1dde68a708f9" (UID: "d6654dc0-fdbd-44b9-adfd-1dde68a708f9"). InnerVolumeSpecName "kube-api-access-rj8zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:04.363429 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.363413 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-maas-api-tls" pod "d6654dc0-fdbd-44b9-adfd-1dde68a708f9" (UID: "d6654dc0-fdbd-44b9-adfd-1dde68a708f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:03:04.462657 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.462614 6107 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-metrics-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:04.462657 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.462657 6107 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-openshift-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:04.462868 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.462672 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-maas-api-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:04.462868 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.462685 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj8zc\" (UniqueName: \"kubernetes.io/projected/d6654dc0-fdbd-44b9-adfd-1dde68a708f9-kube-api-access-rj8zc\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:04.712144 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.712060 6107 generic.go:356] "Generic (PLEG): container finished" podID="d6654dc0-fdbd-44b9-adfd-1dde68a708f9" containerID="f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c" exitCode=0 Oct 31 20:03:04.712144 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.712126 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" Oct 31 20:03:04.712387 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.712131 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" event={"ID":"d6654dc0-fdbd-44b9-adfd-1dde68a708f9","Type":"ContainerDied","Data":"f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c"} Oct 31 20:03:04.712387 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.712266 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96" event={"ID":"d6654dc0-fdbd-44b9-adfd-1dde68a708f9","Type":"ContainerDied","Data":"2a5c779aa7bca40d7578b9ec3107b8e1b590fbdb7f3e1b2134020fbd8f4c5094"} Oct 31 20:03:04.712387 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.712296 6107 scope.go:122] "RemoveContainer" containerID="f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c" Oct 31 20:03:04.724121 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.724099 6107 scope.go:122] "RemoveContainer" containerID="f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c" Oct 31 20:03:04.724408 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:03:04.724391 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c\": container with ID starting with f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c not found: ID does not exist" containerID="f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c" Oct 31 20:03:04.724482 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.724413 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c"} err="failed to get container status \"f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c\": rpc error: code = NotFound desc = could not find container \"f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c\": container with ID starting with f3046c892f7eb4753202d6c4afaa0fbf25458d547422bd704e87208e1d38ce4c not found: ID does not exist" Oct 31 20:03:04.866362 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.866325 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96"] Oct 31 20:03:04.912023 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:04.911981 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-650cf3-85cc969b87-tmn96"] Oct 31 20:03:06.507398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:06.497195 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6654dc0-fdbd-44b9-adfd-1dde68a708f9" path="/var/lib/kubelet/pods/d6654dc0-fdbd-44b9-adfd-1dde68a708f9/volumes" Oct 31 20:03:20.406292 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:03:20.406247 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:03:46.001833 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.001773 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9"] Oct 31 20:03:46.002495 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.002050 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" podUID="8e7c440a-7e60-40a8-a55b-27d8760c990e" containerName="maas-api" containerID="cri-o://3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c" gracePeriod=30 Oct 31 20:03:46.300336 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.300293 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 20:03:46.405306 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.404522 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-maas-api-tls\") pod \"8e7c440a-7e60-40a8-a55b-27d8760c990e\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " Oct 31 20:03:46.405306 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.404715 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-metrics-certs\") pod \"8e7c440a-7e60-40a8-a55b-27d8760c990e\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " Oct 31 20:03:46.405306 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.404791 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7c440a-7e60-40a8-a55b-27d8760c990e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7c440a-7e60-40a8-a55b-27d8760c990e-openshift-service-ca\") pod \"8e7c440a-7e60-40a8-a55b-27d8760c990e\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " Oct 31 20:03:46.405306 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.404852 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7c440a-7e60-40a8-a55b-27d8760c990e-kube-api-access-smwbh\" (UniqueName: \"kubernetes.io/projected/8e7c440a-7e60-40a8-a55b-27d8760c990e-kube-api-access-smwbh\") pod \"8e7c440a-7e60-40a8-a55b-27d8760c990e\" (UID: \"8e7c440a-7e60-40a8-a55b-27d8760c990e\") " Oct 31 20:03:46.413948 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.407312 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7c440a-7e60-40a8-a55b-27d8760c990e-openshift-service-ca" pod "8e7c440a-7e60-40a8-a55b-27d8760c990e" (UID: "8e7c440a-7e60-40a8-a55b-27d8760c990e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:46.415179 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.415084 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7c440a-7e60-40a8-a55b-27d8760c990e-kube-api-access-smwbh" pod "8e7c440a-7e60-40a8-a55b-27d8760c990e" (UID: "8e7c440a-7e60-40a8-a55b-27d8760c990e"). InnerVolumeSpecName "kube-api-access-smwbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:46.416659 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.416612 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-maas-api-tls" pod "8e7c440a-7e60-40a8-a55b-27d8760c990e" (UID: "8e7c440a-7e60-40a8-a55b-27d8760c990e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:03:46.421183 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.421131 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-metrics-certs" pod "8e7c440a-7e60-40a8-a55b-27d8760c990e" (UID: "8e7c440a-7e60-40a8-a55b-27d8760c990e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:03:46.507673 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.507639 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-maas-api-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:46.507673 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.507674 6107 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e7c440a-7e60-40a8-a55b-27d8760c990e-metrics-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:46.507871 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.507687 6107 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7c440a-7e60-40a8-a55b-27d8760c990e-openshift-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:46.507871 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:46.507702 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smwbh\" (UniqueName: \"kubernetes.io/projected/8e7c440a-7e60-40a8-a55b-27d8760c990e-kube-api-access-smwbh\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:47.099726 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:47.099691 6107 generic.go:356] "Generic (PLEG): container finished" podID="8e7c440a-7e60-40a8-a55b-27d8760c990e" containerID="3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c" exitCode=0 Oct 31 20:03:47.100172 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:47.099734 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" Oct 31 20:03:47.100172 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:47.099772 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" event={"ID":"8e7c440a-7e60-40a8-a55b-27d8760c990e","Type":"ContainerDied","Data":"3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c"} Oct 31 20:03:47.100172 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:47.099827 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9" event={"ID":"8e7c440a-7e60-40a8-a55b-27d8760c990e","Type":"ContainerDied","Data":"678ed2fe1d501b1bf76425695e0daed8ec2f033258b772440a0d53d6f59081ad"} Oct 31 20:03:47.100172 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:47.099854 6107 scope.go:122] "RemoveContainer" containerID="3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c" Oct 31 20:03:47.111432 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:47.111406 6107 scope.go:122] "RemoveContainer" containerID="3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c" Oct 31 20:03:47.111749 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:03:47.111727 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c\": container with ID starting with 3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c not found: ID does not exist" containerID="3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c" Oct 31 20:03:47.111829 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:47.111758 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c"} err="failed to get container status \"3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c\": rpc error: code = NotFound desc = could not find container \"3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c\": container with ID starting with 3b3aa16b0d7a8c96d379bd627ce27147400a5004758c4d418d7fab21920e082c not found: ID does not exist" Oct 31 20:03:47.258506 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:47.256390 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9"] Oct 31 20:03:47.259431 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:47.259401 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7cfafd-64d7b96b84-4sjc9"] Oct 31 20:03:48.496596 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:48.496522 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7c440a-7e60-40a8-a55b-27d8760c990e" path="/var/lib/kubelet/pods/8e7c440a-7e60-40a8-a55b-27d8760c990e/volumes" Oct 31 20:03:50.407760 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:03:50.407713 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:03:53.580779 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:53.580729 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m"] Oct 31 20:03:53.581425 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:53.581052 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" podUID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" containerName="istio-proxy" containerID="cri-o://cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256" gracePeriod=30 Oct 31 20:03:55.438707 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:55.438673 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9"] Oct 31 20:03:55.439145 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:55.438967 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" podUID="bdf6ba13-4bc6-48f3-8f9c-535a43d43738" containerName="istio-proxy" containerID="cri-o://e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3" gracePeriod=30 Oct 31 20:03:55.451502 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:55.451473 6107 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:03:55.451617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:55.451537 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" podUID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:03:56.331737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:56.331701 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp"] Oct 31 20:03:56.332042 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:56.331973 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" containerID="cri-o://1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413" gracePeriod=30 Oct 31 20:03:58.835253 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.835224 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 20:03:58.981079 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981048 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-token\" (UniqueName: \"kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-token\") pod \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " Oct 31 20:03:58.981270 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981122 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-envoy\") pod \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " Oct 31 20:03:58.981270 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981215 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-socket\") pod \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " Oct 31 20:03:58.981397 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981307 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-data\") pod \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " Oct 31 20:03:58.981397 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981327 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-certs\") pod \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " Oct 31 20:03:58.981397 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981378 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-podinfo\") pod \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " Oct 31 20:03:58.981628 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981429 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-credential-socket\") pod \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " Oct 31 20:03:58.981628 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981445 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-socket" pod "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" (UID: "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.981628 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981529 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-kube-api-access-5tzqj\" (UniqueName: \"kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-kube-api-access-5tzqj\") pod \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " Oct 31 20:03:58.981628 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981557 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istiod-ca-cert\") pod \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\" (UID: \"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5\") " Oct 31 20:03:58.981628 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981582 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-data" pod "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" (UID: "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.981628 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981613 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-certs" pod "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" (UID: "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.981994 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.981957 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-credential-socket" pod "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" (UID: "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.982059 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.982006 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istiod-ca-cert" pod "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" (UID: "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:58.982263 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.982241 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.982355 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.982269 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.982355 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.982286 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.982355 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.982301 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.982355 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.982317 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.983419 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.983386 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-envoy" pod "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" (UID: "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.983521 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.983460 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-token" pod "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" (UID: "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:58.983629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.983611 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-podinfo" pod "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" (UID: "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:03:58.983707 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:58.983691 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-kube-api-access-5tzqj" pod "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" (UID: "a5ab4f0d-9495-4071-a5d0-13dc5daf25a5"). InnerVolumeSpecName "kube-api-access-5tzqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:59.083586 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.083557 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:59.083586 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.083585 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tzqj\" (UniqueName: \"kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-kube-api-access-5tzqj\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:59.083786 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.083599 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:59.083786 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.083611 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:59.194714 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.194675 6107 generic.go:356] "Generic (PLEG): container finished" podID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" containerID="cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256" exitCode=0 Oct 31 20:03:59.194714 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.194716 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" Oct 31 20:03:59.194986 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.194752 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" event={"ID":"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5","Type":"ContainerDied","Data":"cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256"} Oct 31 20:03:59.194986 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.194804 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m" event={"ID":"a5ab4f0d-9495-4071-a5d0-13dc5daf25a5","Type":"ContainerDied","Data":"4bcd8b282423be6ac411e862c4aa795d0e54fe1fb45e364e813e75301be82e37"} Oct 31 20:03:59.194986 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.194826 6107 scope.go:122] "RemoveContainer" containerID="cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256" Oct 31 20:03:59.205945 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.205913 6107 scope.go:122] "RemoveContainer" containerID="cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256" Oct 31 20:03:59.206205 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:03:59.206188 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256\": container with ID starting with cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256 not found: ID does not exist" containerID="cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256" Oct 31 20:03:59.206273 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.206215 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256"} err="failed to get container status \"cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256\": rpc error: code = NotFound desc = could not find container \"cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256\": container with ID starting with cc17f08b4a2173455bca1f98aa52b24806e1c68641850c7ca57c04ab6411e256 not found: ID does not exist" Oct 31 20:03:59.351253 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.351163 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m"] Oct 31 20:03:59.396932 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:03:59.396891 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-8tm2m"] Oct 31 20:04:00.487999 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.487973 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" path="/var/lib/kubelet/pods/a5ab4f0d-9495-4071-a5d0-13dc5daf25a5/volumes" Oct 31 20:04:00.723054 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.723027 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 20:04:00.808661 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.808623 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istiod-ca-cert\") pod \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " Oct 31 20:04:00.808831 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.808673 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-certs\") pod \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " Oct 31 20:04:00.808831 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.808716 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-envoy\") pod \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " Oct 31 20:04:00.808971 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.808842 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-credential-socket\") pod \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " Oct 31 20:04:00.808971 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.808933 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-token\" (UniqueName: \"kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-token\") pod \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " Oct 31 20:04:00.809098 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809023 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-certs" pod "bdf6ba13-4bc6-48f3-8f9c-535a43d43738" (UID: "bdf6ba13-4bc6-48f3-8f9c-535a43d43738"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:00.809098 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809059 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-kube-api-access-cwhbz\" (UniqueName: \"kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-kube-api-access-cwhbz\") pod \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " Oct 31 20:04:00.809098 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809078 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istiod-ca-cert" pod "bdf6ba13-4bc6-48f3-8f9c-535a43d43738" (UID: "bdf6ba13-4bc6-48f3-8f9c-535a43d43738"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:00.809252 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809188 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-data\") pod \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " Oct 31 20:04:00.809252 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809192 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-credential-socket" pod "bdf6ba13-4bc6-48f3-8f9c-535a43d43738" (UID: "bdf6ba13-4bc6-48f3-8f9c-535a43d43738"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:00.809252 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809222 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-socket\") pod \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " Oct 31 20:04:00.809415 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809256 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-podinfo\") pod \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\" (UID: \"bdf6ba13-4bc6-48f3-8f9c-535a43d43738\") " Oct 31 20:04:00.809415 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809398 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-data" pod "bdf6ba13-4bc6-48f3-8f9c-535a43d43738" (UID: "bdf6ba13-4bc6-48f3-8f9c-535a43d43738"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:00.809526 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809492 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-socket" pod "bdf6ba13-4bc6-48f3-8f9c-535a43d43738" (UID: "bdf6ba13-4bc6-48f3-8f9c-535a43d43738"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:00.809870 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809844 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:00.809870 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809876 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:00.810087 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809907 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:00.810087 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809923 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:00.810087 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.809937 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:00.811333 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.811312 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-kube-api-access-cwhbz" pod "bdf6ba13-4bc6-48f3-8f9c-535a43d43738" (UID: "bdf6ba13-4bc6-48f3-8f9c-535a43d43738"). InnerVolumeSpecName "kube-api-access-cwhbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:00.811444 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.811398 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-podinfo" pod "bdf6ba13-4bc6-48f3-8f9c-535a43d43738" (UID: "bdf6ba13-4bc6-48f3-8f9c-535a43d43738"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:04:00.811578 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.811555 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-envoy" pod "bdf6ba13-4bc6-48f3-8f9c-535a43d43738" (UID: "bdf6ba13-4bc6-48f3-8f9c-535a43d43738"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:00.811691 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.811589 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-token" pod "bdf6ba13-4bc6-48f3-8f9c-535a43d43738" (UID: "bdf6ba13-4bc6-48f3-8f9c-535a43d43738"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:00.911197 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.911168 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:00.911197 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.911194 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwhbz\" (UniqueName: \"kubernetes.io/projected/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-kube-api-access-cwhbz\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:00.911391 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.911209 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:00.911391 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:00.911234 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdf6ba13-4bc6-48f3-8f9c-535a43d43738-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.210307 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.210210 6107 generic.go:356] "Generic (PLEG): container finished" podID="bdf6ba13-4bc6-48f3-8f9c-535a43d43738" containerID="e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3" exitCode=0 Oct 31 20:04:01.210307 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.210267 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" Oct 31 20:04:01.210567 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.210295 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" event={"ID":"bdf6ba13-4bc6-48f3-8f9c-535a43d43738","Type":"ContainerDied","Data":"e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3"} Oct 31 20:04:01.210567 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.210386 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9" event={"ID":"bdf6ba13-4bc6-48f3-8f9c-535a43d43738","Type":"ContainerDied","Data":"b14432e73537251ec167dd8aea1dd8f6b6c0fb32790d527468d9fb2c49509ede"} Oct 31 20:04:01.210567 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.210418 6107 scope.go:122] "RemoveContainer" containerID="e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3" Oct 31 20:04:01.221656 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.221637 6107 scope.go:122] "RemoveContainer" containerID="e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3" Oct 31 20:04:01.222011 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:04:01.221988 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3\": container with ID starting with e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3 not found: ID does not exist" containerID="e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3" Oct 31 20:04:01.222078 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.222021 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3"} err="failed to get container status \"e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3\": rpc error: code = NotFound desc = could not find container \"e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3\": container with ID starting with e37cb5dc40bb5ebfc137d1581deba8bf3ec513057c93426d2e09371d073b51c3 not found: ID does not exist" Oct 31 20:04:01.364126 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.364088 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9"] Oct 31 20:04:01.409486 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.409450 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-8m8d9"] Oct 31 20:04:01.614634 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.614605 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 20:04:01.721073 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.721042 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-data\") pod \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " Oct 31 20:04:01.721272 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.721097 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-podinfo\") pod \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " Oct 31 20:04:01.721272 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.721165 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-socket\") pod \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " Oct 31 20:04:01.721272 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.721218 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-certs\") pod \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " Oct 31 20:04:01.721272 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.721269 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istiod-ca-cert\") pod \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " Oct 31 20:04:01.721508 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.721327 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-credential-socket\") pod \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " Oct 31 20:04:01.721508 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.721356 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-data" pod "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" (UID: "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:01.721508 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.721410 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-token\" (UniqueName: \"kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-token\") pod \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " Oct 31 20:04:01.721701 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.721578 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-envoy\") pod \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " Oct 31 20:04:01.721701 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.721610 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-socket" pod "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" (UID: "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:01.721810 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.721733 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-kube-api-access-55tl6\" (UniqueName: \"kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-kube-api-access-55tl6\") pod \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\" (UID: \"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee\") " Oct 31 20:04:01.722842 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.722805 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-credential-socket" pod "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" (UID: "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:01.723044 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.722868 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istiod-ca-cert" pod "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" (UID: "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:01.723044 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.722932 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.723044 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.722952 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.723219 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.723098 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-certs" pod "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" (UID: "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:01.725438 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.725394 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-envoy" pod "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" (UID: "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:01.727254 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.726461 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-podinfo" pod "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" (UID: "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:04:01.728521 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.728495 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-token" pod "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" (UID: "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:01.729274 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.729248 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-kube-api-access-55tl6" pod "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" (UID: "ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee"). InnerVolumeSpecName "kube-api-access-55tl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:01.824502 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.824460 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.824502 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.824495 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.824502 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.824503 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.824502 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.824513 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.824791 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.824522 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.824791 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.824530 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.824791 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:01.824539 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55tl6\" (UniqueName: \"kubernetes.io/projected/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee-kube-api-access-55tl6\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:02.220330 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:02.220290 6107 generic.go:356] "Generic (PLEG): container finished" podID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerID="1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413" exitCode=0 Oct 31 20:04:02.220511 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:02.220337 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" event={"ID":"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee","Type":"ContainerDied","Data":"1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413"} Oct 31 20:04:02.220511 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:02.220379 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" event={"ID":"ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee","Type":"ContainerDied","Data":"9a75d50b06dbbeefe892f6bc03685be189d9b250fda6bbac19df7a8acb82e357"} Oct 31 20:04:02.220511 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:02.220401 6107 scope.go:122] "RemoveContainer" containerID="1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413" Oct 31 20:04:02.220511 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:02.220342 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp" Oct 31 20:04:02.231092 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:02.231068 6107 scope.go:122] "RemoveContainer" containerID="1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413" Oct 31 20:04:02.231394 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:04:02.231367 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413\": container with ID starting with 1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413 not found: ID does not exist" containerID="1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413" Oct 31 20:04:02.231437 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:02.231404 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413"} err="failed to get container status \"1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413\": rpc error: code = NotFound desc = could not find container \"1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413\": container with ID starting with 1e5e61bb09c4a9abb5045646931fe62aa045e67a207d412845cc84a02c947413 not found: ID does not exist" Oct 31 20:04:02.423042 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:02.423008 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp"] Oct 31 20:04:02.467283 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:02.467249 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-2d7zp"] Oct 31 20:04:02.487863 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:02.487789 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf6ba13-4bc6-48f3-8f9c-535a43d43738" path="/var/lib/kubelet/pods/bdf6ba13-4bc6-48f3-8f9c-535a43d43738/volumes" Oct 31 20:04:02.488379 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:02.488360 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" path="/var/lib/kubelet/pods/ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee/volumes" Oct 31 20:04:07.963322 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.963282 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr"] Oct 31 20:04:07.963918 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.963900 6107 cpu_manager.go:414] "containerMap: removing container" podUID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" containerName="istio-proxy" Oct 31 20:04:07.963991 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.963918 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" containerName="istio-proxy" Oct 31 20:04:07.963991 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.963931 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d6654dc0-fdbd-44b9-adfd-1dde68a708f9" containerName="maas-api" Oct 31 20:04:07.963991 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.963937 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6654dc0-fdbd-44b9-adfd-1dde68a708f9" containerName="maas-api" Oct 31 20:04:07.963991 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.963949 6107 cpu_manager.go:414] "containerMap: removing container" podUID="f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8" containerName="payload-processing" Oct 31 20:04:07.963991 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.963955 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8" containerName="payload-processing" Oct 31 20:04:07.963991 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.963970 6107 cpu_manager.go:414] "containerMap: removing container" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" Oct 31 20:04:07.963991 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.963974 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" Oct 31 20:04:07.963991 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.963983 6107 cpu_manager.go:414] "containerMap: removing container" podUID="bdf6ba13-4bc6-48f3-8f9c-535a43d43738" containerName="istio-proxy" Oct 31 20:04:07.963991 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.963988 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf6ba13-4bc6-48f3-8f9c-535a43d43738" containerName="istio-proxy" Oct 31 20:04:07.964398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.964000 6107 cpu_manager.go:414] "containerMap: removing container" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" Oct 31 20:04:07.964398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.964005 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" Oct 31 20:04:07.964398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.964022 6107 cpu_manager.go:414] "containerMap: removing container" podUID="8e7c440a-7e60-40a8-a55b-27d8760c990e" containerName="maas-api" Oct 31 20:04:07.964398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.964027 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7c440a-7e60-40a8-a55b-27d8760c990e" containerName="maas-api" Oct 31 20:04:07.964398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.964107 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ab4f0d-9495-4071-a5d0-13dc5daf25a5" containerName="istio-proxy" Oct 31 20:04:07.964398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.964116 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6654dc0-fdbd-44b9-adfd-1dde68a708f9" containerName="maas-api" Oct 31 20:04:07.964398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.964123 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="f37adad1-4605-4dfc-b01c-7b9c6b9bb2d8" containerName="payload-processing" Oct 31 20:04:07.964398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.964131 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc2c4e4-0f1c-4eeb-ac5a-a90a6dc0d0ee" containerName="istio-proxy" Oct 31 20:04:07.964398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.964137 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf6ba13-4bc6-48f3-8f9c-535a43d43738" containerName="istio-proxy" Oct 31 20:04:07.964398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.964144 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e2b24f-63f6-40db-aca5-74376cb072f1" containerName="istio-proxy" Oct 31 20:04:07.964398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.964152 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7c440a-7e60-40a8-a55b-27d8760c990e" containerName="maas-api" Oct 31 20:04:07.967574 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:07.967556 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.002269 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.002228 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f0be40-d904-455b-9113-956835869a3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.002452 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.002308 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.002452 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.002400 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.002452 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.002444 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djg9n\" (UniqueName: \"kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-kube-api-access-djg9n\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.002640 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.002473 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.002640 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.002626 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.002749 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.002719 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.002805 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.002745 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f0be40-d904-455b-9113-956835869a3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.002861 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.002807 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.065251 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.065217 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr"] Oct 31 20:04:08.104354 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.104289 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.104630 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.104609 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djg9n\" (UniqueName: \"kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-kube-api-access-djg9n\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.104766 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.104745 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.104931 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.104770 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.104995 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.104946 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.105755 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.105058 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.105755 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.105092 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f0be40-d904-455b-9113-956835869a3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.105755 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.105132 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.105755 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.105163 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.105755 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.105197 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f0be40-d904-455b-9113-956835869a3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.105755 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.105282 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.105755 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.105627 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.105755 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.105706 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.106851 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.106822 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f0be40-d904-455b-9113-956835869a3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.107980 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.107954 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f0be40-d904-455b-9113-956835869a3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.108087 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.108014 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.159700 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.159665 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.159870 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.159787 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djg9n\" (UniqueName: \"kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-kube-api-access-djg9n\") pod \"maas-default-gateway-openshift-default-58d898bd55-xcrlr\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.282358 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.282321 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:08.700599 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.700564 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr"] Oct 31 20:04:08.702489 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:04:08.702457 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f0be40_d904_455b_9113_956835869a3d.slice/crio-e441e77723bbf124e0ca37813d6f2d3af8aa36069bcf28329c985c94c16541b6 WatchSource:0}: Error finding container e441e77723bbf124e0ca37813d6f2d3af8aa36069bcf28329c985c94c16541b6: Status 404 returned error can't find the container with id e441e77723bbf124e0ca37813d6f2d3af8aa36069bcf28329c985c94c16541b6 Oct 31 20:04:08.705200 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.705167 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:04:08.705308 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.705251 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:04:08.705308 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:08.705294 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:04:09.272093 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:09.272036 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" event={"ID":"46f0be40-d904-455b-9113-956835869a3d","Type":"ContainerStarted","Data":"7b49a81d025c4710481f551d04ce0fb93739200510b3250db0105ac183d7d549"} Oct 31 20:04:09.272473 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:09.272135 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" event={"ID":"46f0be40-d904-455b-9113-956835869a3d","Type":"ContainerStarted","Data":"e441e77723bbf124e0ca37813d6f2d3af8aa36069bcf28329c985c94c16541b6"} Oct 31 20:04:09.285958 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:09.285919 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:09.288646 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:09.287858 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xcrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 20:04:09.288646 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:09.287944 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" podUID="46f0be40-d904-455b-9113-956835869a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 20:04:09.473659 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:09.473602 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" podStartSLOduration=2.473587067 podStartE2EDuration="2.473587067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:09.432614454 +0000 UTC m=+2821.066008362" watchObservedRunningTime="2026-10-31 20:04:09.473587067 +0000 UTC m=+2821.106980999" Oct 31 20:04:10.283340 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:10.283302 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xcrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 20:04:10.283833 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:10.283379 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" podUID="46f0be40-d904-455b-9113-956835869a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 20:04:11.283691 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:11.283659 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xcrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 20:04:11.284117 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:11.283704 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" podUID="46f0be40-d904-455b-9113-956835869a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 20:04:12.283177 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:12.283039 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xcrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 20:04:12.283177 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:12.283093 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" podUID="46f0be40-d904-455b-9113-956835869a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 20:04:12.726981 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:12.719929 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c"] Oct 31 20:04:12.837251 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:12.837214 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck"] Oct 31 20:04:13.282871 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:13.282836 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xcrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 20:04:13.283090 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:13.282903 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" podUID="46f0be40-d904-455b-9113-956835869a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 20:04:14.287903 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:14.287361 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xcrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 20:04:14.287903 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:14.287444 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" podUID="46f0be40-d904-455b-9113-956835869a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 20:04:15.283144 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:15.283102 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xcrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 20:04:15.283332 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:15.283162 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" podUID="46f0be40-d904-455b-9113-956835869a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 20:04:16.283574 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:16.283533 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xcrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 20:04:16.284010 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:16.283612 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" podUID="46f0be40-d904-455b-9113-956835869a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 20:04:17.287374 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:17.287341 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:17.287858 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:17.287694 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:17.288538 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:17.288516 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:04:17.720739 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:17.720655 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" podUID="54065907-30f3-4605-9bf1-e7e9174af9a7" containerName="payload-processing" containerID="cri-o://14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051" gracePeriod=25 Oct 31 20:04:17.838651 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:17.838583 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" podUID="a9933276-a77f-4e32-8a01-68df35b1813c" containerName="payload-pre-processing" containerID="cri-o://a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830" gracePeriod=25 Oct 31 20:04:18.013130 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.013100 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" Oct 31 20:04:18.117587 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.117562 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" Oct 31 20:04:18.147688 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.147646 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54065907-30f3-4605-9bf1-e7e9174af9a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54065907-30f3-4605-9bf1-e7e9174af9a7-plugins-config-volume\") pod \"54065907-30f3-4605-9bf1-e7e9174af9a7\" (UID: \"54065907-30f3-4605-9bf1-e7e9174af9a7\") " Oct 31 20:04:18.147912 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.147766 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54065907-30f3-4605-9bf1-e7e9174af9a7-kube-api-access-sj922\" (UniqueName: \"kubernetes.io/projected/54065907-30f3-4605-9bf1-e7e9174af9a7-kube-api-access-sj922\") pod \"54065907-30f3-4605-9bf1-e7e9174af9a7\" (UID: \"54065907-30f3-4605-9bf1-e7e9174af9a7\") " Oct 31 20:04:18.148430 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.148397 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54065907-30f3-4605-9bf1-e7e9174af9a7-plugins-config-volume" pod "54065907-30f3-4605-9bf1-e7e9174af9a7" (UID: "54065907-30f3-4605-9bf1-e7e9174af9a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:18.150716 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.150689 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54065907-30f3-4605-9bf1-e7e9174af9a7-kube-api-access-sj922" pod "54065907-30f3-4605-9bf1-e7e9174af9a7" (UID: "54065907-30f3-4605-9bf1-e7e9174af9a7"). InnerVolumeSpecName "kube-api-access-sj922". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:18.249148 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.249059 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9933276-a77f-4e32-8a01-68df35b1813c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9933276-a77f-4e32-8a01-68df35b1813c-plugins-config-volume\") pod \"a9933276-a77f-4e32-8a01-68df35b1813c\" (UID: \"a9933276-a77f-4e32-8a01-68df35b1813c\") " Oct 31 20:04:18.249148 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.249111 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9933276-a77f-4e32-8a01-68df35b1813c-kube-api-access-ctdjw\" (UniqueName: \"kubernetes.io/projected/a9933276-a77f-4e32-8a01-68df35b1813c-kube-api-access-ctdjw\") pod \"a9933276-a77f-4e32-8a01-68df35b1813c\" (UID: \"a9933276-a77f-4e32-8a01-68df35b1813c\") " Oct 31 20:04:18.249507 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.249482 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9933276-a77f-4e32-8a01-68df35b1813c-plugins-config-volume" pod "a9933276-a77f-4e32-8a01-68df35b1813c" (UID: "a9933276-a77f-4e32-8a01-68df35b1813c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:18.250193 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.249846 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54065907-30f3-4605-9bf1-e7e9174af9a7-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:18.250193 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.249874 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9933276-a77f-4e32-8a01-68df35b1813c-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:18.250193 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.249918 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj922\" (UniqueName: \"kubernetes.io/projected/54065907-30f3-4605-9bf1-e7e9174af9a7-kube-api-access-sj922\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:18.251489 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.251456 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9933276-a77f-4e32-8a01-68df35b1813c-kube-api-access-ctdjw" pod "a9933276-a77f-4e32-8a01-68df35b1813c" (UID: "a9933276-a77f-4e32-8a01-68df35b1813c"). InnerVolumeSpecName "kube-api-access-ctdjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:18.350987 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.350928 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdjw\" (UniqueName: \"kubernetes.io/projected/a9933276-a77f-4e32-8a01-68df35b1813c-kube-api-access-ctdjw\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:18.370999 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.370966 6107 generic.go:356] "Generic (PLEG): container finished" podID="a9933276-a77f-4e32-8a01-68df35b1813c" containerID="a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830" exitCode=0 Oct 31 20:04:18.371176 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.371013 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" Oct 31 20:04:18.371176 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.371059 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" event={"ID":"a9933276-a77f-4e32-8a01-68df35b1813c","Type":"ContainerDied","Data":"a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830"} Oct 31 20:04:18.371176 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.371111 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck" event={"ID":"a9933276-a77f-4e32-8a01-68df35b1813c","Type":"ContainerDied","Data":"619d062de535088c3f033de68a90793f6351f58ea8d9cf3b0af091279e6dc739"} Oct 31 20:04:18.371176 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.371132 6107 scope.go:122] "RemoveContainer" containerID="a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830" Oct 31 20:04:18.372694 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.372673 6107 generic.go:356] "Generic (PLEG): container finished" podID="54065907-30f3-4605-9bf1-e7e9174af9a7" containerID="14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051" exitCode=0 Oct 31 20:04:18.372774 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.372713 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" Oct 31 20:04:18.372813 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.372766 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" event={"ID":"54065907-30f3-4605-9bf1-e7e9174af9a7","Type":"ContainerDied","Data":"14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051"} Oct 31 20:04:18.372861 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.372822 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c" event={"ID":"54065907-30f3-4605-9bf1-e7e9174af9a7","Type":"ContainerDied","Data":"39b8bec3cbe255976488a9a6ebaf0a5f7a62aea51321ce28b8d32d88335a7717"} Oct 31 20:04:18.381399 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.381380 6107 scope.go:122] "RemoveContainer" containerID="a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830" Oct 31 20:04:18.381735 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:04:18.381713 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830\": container with ID starting with a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830 not found: ID does not exist" containerID="a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830" Oct 31 20:04:18.381833 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.381743 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830"} err="failed to get container status \"a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830\": rpc error: code = NotFound desc = could not find container \"a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830\": container with ID starting with a3d3b47671c71b72d60e29736ce1568ca77ee99f31f669cc89ab21679d349830 not found: ID does not exist" Oct 31 20:04:18.381833 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.381763 6107 scope.go:122] "RemoveContainer" containerID="14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051" Oct 31 20:04:18.391872 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.391855 6107 scope.go:122] "RemoveContainer" containerID="14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051" Oct 31 20:04:18.392181 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:04:18.392160 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051\": container with ID starting with 14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051 not found: ID does not exist" containerID="14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051" Oct 31 20:04:18.392225 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.392191 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051"} err="failed to get container status \"14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051\": rpc error: code = NotFound desc = could not find container \"14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051\": container with ID starting with 14bd9d07f7019718ab4c2a868a903c445ecadbed7834baf782a9480c713a9051 not found: ID does not exist" Oct 31 20:04:18.523299 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.523156 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck"] Oct 31 20:04:18.526729 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.526701 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13e9b4-c5b654f6c-mqfck"] Oct 31 20:04:18.633168 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.633135 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c"] Oct 31 20:04:18.634604 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:18.634572 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13e9b4-55b44577df-wq26c"] Oct 31 20:04:20.407105 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:04:20.407063 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:04:20.486938 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:20.486877 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54065907-30f3-4605-9bf1-e7e9174af9a7" path="/var/lib/kubelet/pods/54065907-30f3-4605-9bf1-e7e9174af9a7/volumes" Oct 31 20:04:20.487385 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:04:20.487370 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9933276-a77f-4e32-8a01-68df35b1813c" path="/var/lib/kubelet/pods/a9933276-a77f-4e32-8a01-68df35b1813c/volumes" Oct 31 20:04:50.404480 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:04:50.404441 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:05:07.320168 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.320127 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488"] Oct 31 20:05:07.320774 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.320334 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" podUID="fa28e736-5f55-4166-9457-4390fce20842" containerName="maas-api" containerID="cri-o://5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab" gracePeriod=30 Oct 31 20:05:07.601776 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.601750 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:05:07.722303 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.722267 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-maas-api-tls\") pod \"fa28e736-5f55-4166-9457-4390fce20842\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " Oct 31 20:05:07.722518 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.722425 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa28e736-5f55-4166-9457-4390fce20842-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa28e736-5f55-4166-9457-4390fce20842-openshift-service-ca\") pod \"fa28e736-5f55-4166-9457-4390fce20842\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " Oct 31 20:05:07.722518 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.722512 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa28e736-5f55-4166-9457-4390fce20842-kube-api-access-p97f4\" (UniqueName: \"kubernetes.io/projected/fa28e736-5f55-4166-9457-4390fce20842-kube-api-access-p97f4\") pod \"fa28e736-5f55-4166-9457-4390fce20842\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " Oct 31 20:05:07.723140 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.722614 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-metrics-certs\") pod \"fa28e736-5f55-4166-9457-4390fce20842\" (UID: \"fa28e736-5f55-4166-9457-4390fce20842\") " Oct 31 20:05:07.723140 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.722958 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa28e736-5f55-4166-9457-4390fce20842-openshift-service-ca" pod "fa28e736-5f55-4166-9457-4390fce20842" (UID: "fa28e736-5f55-4166-9457-4390fce20842"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:07.723443 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.723363 6107 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa28e736-5f55-4166-9457-4390fce20842-openshift-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:07.742269 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.741647 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-maas-api-tls" pod "fa28e736-5f55-4166-9457-4390fce20842" (UID: "fa28e736-5f55-4166-9457-4390fce20842"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:07.754640 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.754111 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa28e736-5f55-4166-9457-4390fce20842-kube-api-access-p97f4" pod "fa28e736-5f55-4166-9457-4390fce20842" (UID: "fa28e736-5f55-4166-9457-4390fce20842"). InnerVolumeSpecName "kube-api-access-p97f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:07.764391 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.764098 6107 generic.go:356] "Generic (PLEG): container finished" podID="fa28e736-5f55-4166-9457-4390fce20842" containerID="5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab" exitCode=0 Oct 31 20:05:07.764391 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.764234 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" event={"ID":"fa28e736-5f55-4166-9457-4390fce20842","Type":"ContainerDied","Data":"5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab"} Oct 31 20:05:07.764391 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.764285 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" event={"ID":"fa28e736-5f55-4166-9457-4390fce20842","Type":"ContainerDied","Data":"33c6d31043020afe529e890c4faf881d6b9f5f1ecd1605b741099292def48bf9"} Oct 31 20:05:07.764391 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.764312 6107 scope.go:122] "RemoveContainer" containerID="5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab" Oct 31 20:05:07.764674 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.764482 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488" Oct 31 20:05:07.765241 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.765210 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-metrics-certs" pod "fa28e736-5f55-4166-9457-4390fce20842" (UID: "fa28e736-5f55-4166-9457-4390fce20842"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:07.783581 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.783547 6107 scope.go:122] "RemoveContainer" containerID="5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab" Oct 31 20:05:07.783963 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:05:07.783941 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab\": container with ID starting with 5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab not found: ID does not exist" containerID="5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab" Oct 31 20:05:07.784099 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.783972 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab"} err="failed to get container status \"5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab\": rpc error: code = NotFound desc = could not find container \"5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab\": container with ID starting with 5c97f72a44635abda8574763eb6f126ec12ff050a3997b21cf8b86ea68db4cab not found: ID does not exist" Oct 31 20:05:07.824429 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.824399 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p97f4\" (UniqueName: \"kubernetes.io/projected/fa28e736-5f55-4166-9457-4390fce20842-kube-api-access-p97f4\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:07.824429 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.824430 6107 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-metrics-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:07.824638 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:07.824441 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa28e736-5f55-4166-9457-4390fce20842-maas-api-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:08.177274 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:08.177196 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488"] Oct 31 20:05:08.181206 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:08.181178 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-13e9b4-5559788fd-xr488"] Oct 31 20:05:08.487959 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:08.487931 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa28e736-5f55-4166-9457-4390fce20842" path="/var/lib/kubelet/pods/fa28e736-5f55-4166-9457-4390fce20842/volumes" Oct 31 20:05:16.123858 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:16.123822 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4"] Oct 31 20:05:16.124458 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:16.124066 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" containerID="cri-o://b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f" gracePeriod=30 Oct 31 20:05:20.378825 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:20.378714 6107 patch_prober.go:29] interesting pod/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:05:20.379345 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:20.378816 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:05:20.404007 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:05:20.403971 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:05:21.389507 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.389436 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 20:05:21.408648 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.408618 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-kube-api-access-hwxlb\" (UniqueName: \"kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-kube-api-access-hwxlb\") pod \"d2824e53-a817-4149-b6e4-49453b7c3d5b\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " Oct 31 20:05:21.408787 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.408764 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-data\") pod \"d2824e53-a817-4149-b6e4-49453b7c3d5b\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " Oct 31 20:05:21.408848 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.408800 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-token\" (UniqueName: \"kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-token\") pod \"d2824e53-a817-4149-b6e4-49453b7c3d5b\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " Oct 31 20:05:21.408848 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.408834 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-socket\") pod \"d2824e53-a817-4149-b6e4-49453b7c3d5b\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " Oct 31 20:05:21.408990 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.408904 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-envoy\") pod \"d2824e53-a817-4149-b6e4-49453b7c3d5b\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " Oct 31 20:05:21.408990 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.408948 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-credential-socket\") pod \"d2824e53-a817-4149-b6e4-49453b7c3d5b\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " Oct 31 20:05:21.409093 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.408995 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-certs\") pod \"d2824e53-a817-4149-b6e4-49453b7c3d5b\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " Oct 31 20:05:21.409093 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.409037 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-podinfo\") pod \"d2824e53-a817-4149-b6e4-49453b7c3d5b\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " Oct 31 20:05:21.409093 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.409078 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2824e53-a817-4149-b6e4-49453b7c3d5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2824e53-a817-4149-b6e4-49453b7c3d5b-istiod-ca-cert\") pod \"d2824e53-a817-4149-b6e4-49453b7c3d5b\" (UID: \"d2824e53-a817-4149-b6e4-49453b7c3d5b\") " Oct 31 20:05:21.409552 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.409331 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-data" pod "d2824e53-a817-4149-b6e4-49453b7c3d5b" (UID: "d2824e53-a817-4149-b6e4-49453b7c3d5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.409762 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.409738 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.409875 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.409710 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2824e53-a817-4149-b6e4-49453b7c3d5b-istiod-ca-cert" pod "d2824e53-a817-4149-b6e4-49453b7c3d5b" (UID: "d2824e53-a817-4149-b6e4-49453b7c3d5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:21.409989 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.409966 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-socket" pod "d2824e53-a817-4149-b6e4-49453b7c3d5b" (UID: "d2824e53-a817-4149-b6e4-49453b7c3d5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.410047 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.409869 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-certs" pod "d2824e53-a817-4149-b6e4-49453b7c3d5b" (UID: "d2824e53-a817-4149-b6e4-49453b7c3d5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.410098 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.410060 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-credential-socket" pod "d2824e53-a817-4149-b6e4-49453b7c3d5b" (UID: "d2824e53-a817-4149-b6e4-49453b7c3d5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.411760 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.411714 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-kube-api-access-hwxlb" pod "d2824e53-a817-4149-b6e4-49453b7c3d5b" (UID: "d2824e53-a817-4149-b6e4-49453b7c3d5b"). InnerVolumeSpecName "kube-api-access-hwxlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:21.411939 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.411725 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-envoy" pod "d2824e53-a817-4149-b6e4-49453b7c3d5b" (UID: "d2824e53-a817-4149-b6e4-49453b7c3d5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.412078 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.412056 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-token" pod "d2824e53-a817-4149-b6e4-49453b7c3d5b" (UID: "d2824e53-a817-4149-b6e4-49453b7c3d5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:21.412078 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.412069 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-podinfo" pod "d2824e53-a817-4149-b6e4-49453b7c3d5b" (UID: "d2824e53-a817-4149-b6e4-49453b7c3d5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:05:21.511449 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.511416 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.511449 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.511443 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.511449 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.511452 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.511449 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.511460 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2824e53-a817-4149-b6e4-49453b7c3d5b-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.511776 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.511470 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwxlb\" (UniqueName: \"kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-kube-api-access-hwxlb\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.511776 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.511482 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.511776 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.511493 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.511776 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.511501 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2824e53-a817-4149-b6e4-49453b7c3d5b-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.607385 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.607350 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9"] Oct 31 20:05:21.624469 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.624435 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm"] Oct 31 20:05:21.868268 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.868227 6107 generic.go:356] "Generic (PLEG): container finished" podID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerID="b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f" exitCode=0 Oct 31 20:05:21.868469 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.868283 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" Oct 31 20:05:21.868469 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.868317 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" event={"ID":"d2824e53-a817-4149-b6e4-49453b7c3d5b","Type":"ContainerDied","Data":"b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f"} Oct 31 20:05:21.868469 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.868373 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4" event={"ID":"d2824e53-a817-4149-b6e4-49453b7c3d5b","Type":"ContainerDied","Data":"ccd2341ced0701b965e2cca435ce225ca7b76fc572f2c80071c7770e8d75bc4c"} Oct 31 20:05:21.868469 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.868401 6107 scope.go:122] "RemoveContainer" containerID="b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f" Oct 31 20:05:21.879013 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.878994 6107 scope.go:122] "RemoveContainer" containerID="ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273" Oct 31 20:05:21.888908 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.888846 6107 scope.go:122] "RemoveContainer" containerID="b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f" Oct 31 20:05:21.889212 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:05:21.889191 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f\": container with ID starting with b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f not found: ID does not exist" containerID="b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f" Oct 31 20:05:21.889321 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.889222 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f"} err="failed to get container status \"b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f\": rpc error: code = NotFound desc = could not find container \"b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f\": container with ID starting with b50417cf519a41d3fe65003e199c6bb2dd51b340cbda8fb05b6241676b8fbf6f not found: ID does not exist" Oct 31 20:05:21.889321 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.889250 6107 scope.go:122] "RemoveContainer" containerID="ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273" Oct 31 20:05:21.889506 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:05:21.889482 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273\": container with ID starting with ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273 not found: ID does not exist" containerID="ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273" Oct 31 20:05:21.889553 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:21.889515 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273"} err="failed to get container status \"ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273\": rpc error: code = NotFound desc = could not find container \"ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273\": container with ID starting with ef9a1ce2a52cf78c2b802bafdc8bf261eeacdc525c2c39880024542fee7ea273 not found: ID does not exist" Oct 31 20:05:22.022296 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:22.022260 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4"] Oct 31 20:05:22.028309 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:22.028274 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-13e9b4-openshift-default-bd88db87f-s2zq4"] Oct 31 20:05:22.487671 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:22.487642 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" path="/var/lib/kubelet/pods/d2824e53-a817-4149-b6e4-49453b7c3d5b/volumes" Oct 31 20:05:28.690584 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:28.690554 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl"] Oct 31 20:05:31.333959 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:31.333926 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl"] Oct 31 20:05:33.691421 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:33.691380 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" podUID="8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35" containerName="payload-processing" containerID="cri-o://7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f" gracePeriod=25 Oct 31 20:05:35.554493 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.554430 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6"] Oct 31 20:05:35.555211 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555188 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" Oct 31 20:05:35.555289 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555213 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" Oct 31 20:05:35.555344 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555296 6107 cpu_manager.go:414] "containerMap: removing container" podUID="54065907-30f3-4605-9bf1-e7e9174af9a7" containerName="payload-processing" Oct 31 20:05:35.555344 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555304 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54065907-30f3-4605-9bf1-e7e9174af9a7" containerName="payload-processing" Oct 31 20:05:35.555344 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555331 6107 cpu_manager.go:414] "containerMap: removing container" podUID="a9933276-a77f-4e32-8a01-68df35b1813c" containerName="payload-pre-processing" Oct 31 20:05:35.555344 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555338 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9933276-a77f-4e32-8a01-68df35b1813c" containerName="payload-pre-processing" Oct 31 20:05:35.555549 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555352 6107 cpu_manager.go:414] "containerMap: removing container" podUID="fa28e736-5f55-4166-9457-4390fce20842" containerName="maas-api" Oct 31 20:05:35.555549 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555359 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa28e736-5f55-4166-9457-4390fce20842" containerName="maas-api" Oct 31 20:05:35.555549 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555495 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" Oct 31 20:05:35.555549 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555507 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="54065907-30f3-4605-9bf1-e7e9174af9a7" containerName="payload-processing" Oct 31 20:05:35.555549 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555518 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9933276-a77f-4e32-8a01-68df35b1813c" containerName="payload-pre-processing" Oct 31 20:05:35.555549 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555527 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa28e736-5f55-4166-9457-4390fce20842" containerName="maas-api" Oct 31 20:05:35.555549 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.555536 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" Oct 31 20:05:35.560949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.560928 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" Oct 31 20:05:35.658911 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.637806 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6"] Oct 31 20:05:35.689795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.659232 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ef44dd\"" Oct 31 20:05:35.689795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.660459 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ef44dd-dockercfg-dwnrp\"" Oct 31 20:05:35.689795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.669187 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq"] Oct 31 20:05:35.689795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.670537 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" Oct 31 20:05:35.689795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.670558 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2824e53-a817-4149-b6e4-49453b7c3d5b" containerName="istio-proxy" Oct 31 20:05:35.689795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.680585 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6\" (UID: \"784d6b85-dad0-4fb8-a9cd-153ec9f270d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" Oct 31 20:05:35.689795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.680749 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn6r7\" (UniqueName: \"kubernetes.io/projected/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-kube-api-access-wn6r7\") pod \"payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6\" (UID: \"784d6b85-dad0-4fb8-a9cd-153ec9f270d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" Oct 31 20:05:35.697322 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.697293 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" Oct 31 20:05:35.728036 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.728003 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq"] Oct 31 20:05:35.781704 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.781674 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvgzj\" (UniqueName: \"kubernetes.io/projected/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-kube-api-access-qvgzj\") pod \"payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq\" (UID: \"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" Oct 31 20:05:35.781909 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.781720 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq\" (UID: \"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" Oct 31 20:05:35.781909 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.781853 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6\" (UID: \"784d6b85-dad0-4fb8-a9cd-153ec9f270d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" Oct 31 20:05:35.782043 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.782028 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6r7\" (UniqueName: \"kubernetes.io/projected/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-kube-api-access-wn6r7\") pod \"payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6\" (UID: \"784d6b85-dad0-4fb8-a9cd-153ec9f270d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" Oct 31 20:05:35.782599 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.782581 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6\" (UID: \"784d6b85-dad0-4fb8-a9cd-153ec9f270d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" Oct 31 20:05:35.836128 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.836049 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6r7\" (UniqueName: \"kubernetes.io/projected/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-kube-api-access-wn6r7\") pod \"payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6\" (UID: \"784d6b85-dad0-4fb8-a9cd-153ec9f270d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" Oct 31 20:05:35.875272 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.875243 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" Oct 31 20:05:35.882949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.882923 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq\" (UID: \"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" Oct 31 20:05:35.883200 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.883180 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvgzj\" (UniqueName: \"kubernetes.io/projected/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-kube-api-access-qvgzj\") pod \"payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq\" (UID: \"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" Oct 31 20:05:35.883640 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.883622 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq\" (UID: \"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" Oct 31 20:05:35.954156 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:35.951620 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvgzj\" (UniqueName: \"kubernetes.io/projected/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-kube-api-access-qvgzj\") pod \"payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq\" (UID: \"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" Oct 31 20:05:36.020317 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:36.020286 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" Oct 31 20:05:36.243868 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:36.243844 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6"] Oct 31 20:05:36.245459 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:05:36.245430 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784d6b85_dad0_4fb8_a9cd_153ec9f270d2.slice/crio-c575e9162e8dad58271ca3297e921972c04677ebcf233f986dfe69eeaf4cb2ff WatchSource:0}: Error finding container c575e9162e8dad58271ca3297e921972c04677ebcf233f986dfe69eeaf4cb2ff: Status 404 returned error can't find the container with id c575e9162e8dad58271ca3297e921972c04677ebcf233f986dfe69eeaf4cb2ff Oct 31 20:05:36.334224 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:36.334191 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" podUID="801671a2-4d7f-4734-8a10-1d7975dda195" containerName="payload-pre-processing" containerID="cri-o://cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80" gracePeriod=25 Oct 31 20:05:36.387702 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:36.387669 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq"] Oct 31 20:05:36.389619 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:05:36.389591 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed8c1b4_a002_4087_8539_fd8a7c1ce7d3.slice/crio-d084220e4fe6c0372ade0113d94823f3991d80aee4934fa02851ce20f4f009ae WatchSource:0}: Error finding container d084220e4fe6c0372ade0113d94823f3991d80aee4934fa02851ce20f4f009ae: Status 404 returned error can't find the container with id d084220e4fe6c0372ade0113d94823f3991d80aee4934fa02851ce20f4f009ae Oct 31 20:05:36.630266 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:36.630143 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" containerID="cri-o://0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92" gracePeriod=45 Oct 31 20:05:36.645918 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:36.645854 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" containerID="cri-o://947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913" gracePeriod=45 Oct 31 20:05:36.946924 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:36.945175 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 20:05:37.012632 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.011429 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 20:05:37.016930 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.016761 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" event={"ID":"784d6b85-dad0-4fb8-a9cd-153ec9f270d2","Type":"ContainerStarted","Data":"5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466"} Oct 31 20:05:37.017245 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.017218 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" event={"ID":"784d6b85-dad0-4fb8-a9cd-153ec9f270d2","Type":"ContainerStarted","Data":"c575e9162e8dad58271ca3297e921972c04677ebcf233f986dfe69eeaf4cb2ff"} Oct 31 20:05:37.017313 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.017265 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" Oct 31 20:05:37.018678 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.018643 6107 generic.go:356] "Generic (PLEG): container finished" podID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerID="947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913" exitCode=0 Oct 31 20:05:37.018795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.018680 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" Oct 31 20:05:37.018795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.018715 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" event={"ID":"e43bb0c6-0f01-4106-99f8-7a2c8985315f","Type":"ContainerDied","Data":"947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913"} Oct 31 20:05:37.018795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.018750 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" event={"ID":"e43bb0c6-0f01-4106-99f8-7a2c8985315f","Type":"ContainerDied","Data":"14cb34ddbed40fe10b414937744b82de7dc9a57e7d06970dc6046940cb7c0b16"} Oct 31 20:05:37.018795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.018769 6107 scope.go:122] "RemoveContainer" containerID="947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913" Oct 31 20:05:37.021722 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.021698 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" event={"ID":"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3","Type":"ContainerStarted","Data":"ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f"} Oct 31 20:05:37.021863 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.021755 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" Oct 31 20:05:37.021863 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.021781 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" event={"ID":"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3","Type":"ContainerStarted","Data":"d084220e4fe6c0372ade0113d94823f3991d80aee4934fa02851ce20f4f009ae"} Oct 31 20:05:37.023117 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.023091 6107 generic.go:356] "Generic (PLEG): container finished" podID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerID="0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92" exitCode=0 Oct 31 20:05:37.023242 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.023226 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" event={"ID":"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf","Type":"ContainerDied","Data":"0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92"} Oct 31 20:05:37.023304 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.023258 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" event={"ID":"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf","Type":"ContainerDied","Data":"f7dcbc30485f92adbacedbba2e573b7f7035c32fb23a60a6576ba48b30d715dc"} Oct 31 20:05:37.023360 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.023316 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" Oct 31 20:05:37.034110 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.034090 6107 scope.go:122] "RemoveContainer" containerID="947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913" Oct 31 20:05:37.034500 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:05:37.034471 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913\": container with ID starting with 947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913 not found: ID does not exist" containerID="947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913" Oct 31 20:05:37.034561 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.034514 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913"} err="failed to get container status \"947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913\": rpc error: code = NotFound desc = could not find container \"947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913\": container with ID starting with 947e64f9ceba9ed84ac18a357524addc4e35ea549064353ae7672944352b7913 not found: ID does not exist" Oct 31 20:05:37.034561 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.034538 6107 scope.go:122] "RemoveContainer" containerID="0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92" Oct 31 20:05:37.046175 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.046145 6107 scope.go:122] "RemoveContainer" containerID="0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92" Oct 31 20:05:37.046677 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:05:37.046654 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92\": container with ID starting with 0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92 not found: ID does not exist" containerID="0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92" Oct 31 20:05:37.046759 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.046689 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92"} err="failed to get container status \"0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92\": rpc error: code = NotFound desc = could not find container \"0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92\": container with ID starting with 0e1fbffade7f264b82bc3973b70676ff044e9b0598b276b9bf619f82d5cadc92 not found: ID does not exist" Oct 31 20:05:37.104079 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.104042 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-home\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-home\") pod \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " Oct 31 20:05:37.104324 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.104300 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-home" pod "0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" (UID: "0a9b87de-cba8-4ff6-ab19-9e1ce089feaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.104618 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.104586 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tls-certs\") pod \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " Oct 31 20:05:37.104708 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.104657 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tmp-dir\") pod \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " Oct 31 20:05:37.104708 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.104695 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tmp-dir\") pod \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " Oct 31 20:05:37.104812 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.104717 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-dshm\") pod \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " Oct 31 20:05:37.104812 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.104757 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-kube-api-access-2xsfn\" (UniqueName: \"kubernetes.io/projected/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-kube-api-access-2xsfn\") pod \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " Oct 31 20:05:37.104812 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.104796 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-model-cache\") pod \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\" (UID: \"0a9b87de-cba8-4ff6-ab19-9e1ce089feaf\") " Oct 31 20:05:37.104997 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.104827 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-home\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-home\") pod \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " Oct 31 20:05:37.104997 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.104970 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tmp-dir" pod "0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" (UID: "0a9b87de-cba8-4ff6-ab19-9e1ce089feaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.104997 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.104983 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43bb0c6-0f01-4106-99f8-7a2c8985315f-kube-api-access-7tl25\" (UniqueName: \"kubernetes.io/projected/e43bb0c6-0f01-4106-99f8-7a2c8985315f-kube-api-access-7tl25\") pod \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " Oct 31 20:05:37.105155 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.105024 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-model-cache\") pod \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " Oct 31 20:05:37.105473 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.105447 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tmp-dir" pod "e43bb0c6-0f01-4106-99f8-7a2c8985315f" (UID: "e43bb0c6-0f01-4106-99f8-7a2c8985315f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.105695 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.105677 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.105775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.105702 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.105775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.105718 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.105964 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.105941 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-model-cache" pod "0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" (UID: "0a9b87de-cba8-4ff6-ab19-9e1ce089feaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.106159 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.106137 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-home" pod "e43bb0c6-0f01-4106-99f8-7a2c8985315f" (UID: "e43bb0c6-0f01-4106-99f8-7a2c8985315f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.106297 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.106276 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-model-cache" pod "e43bb0c6-0f01-4106-99f8-7a2c8985315f" (UID: "e43bb0c6-0f01-4106-99f8-7a2c8985315f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.107449 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.107414 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tls-certs" pod "0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" (UID: "0a9b87de-cba8-4ff6-ab19-9e1ce089feaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:37.108078 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.108042 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-kube-api-access-2xsfn" pod "0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" (UID: "0a9b87de-cba8-4ff6-ab19-9e1ce089feaf"). InnerVolumeSpecName "kube-api-access-2xsfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:37.108719 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.108695 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-dshm" pod "0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" (UID: "0a9b87de-cba8-4ff6-ab19-9e1ce089feaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.109013 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.108993 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43bb0c6-0f01-4106-99f8-7a2c8985315f-kube-api-access-7tl25" pod "e43bb0c6-0f01-4106-99f8-7a2c8985315f" (UID: "e43bb0c6-0f01-4106-99f8-7a2c8985315f"). InnerVolumeSpecName "kube-api-access-7tl25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:37.206712 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.206627 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-dshm\") pod \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " Oct 31 20:05:37.206936 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.206857 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tls-certs\" (UniqueName: \"kubernetes.io/secret/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tls-certs\") pod \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\" (UID: \"e43bb0c6-0f01-4106-99f8-7a2c8985315f\") " Oct 31 20:05:37.207285 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.207265 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.207369 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.207288 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.207369 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.207303 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tl25\" (UniqueName: \"kubernetes.io/projected/e43bb0c6-0f01-4106-99f8-7a2c8985315f-kube-api-access-7tl25\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.207369 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.207318 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.207369 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.207331 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.207369 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.207343 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.207369 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.207356 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xsfn\" (UniqueName: \"kubernetes.io/projected/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf-kube-api-access-2xsfn\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.208774 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.208752 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-dshm" pod "e43bb0c6-0f01-4106-99f8-7a2c8985315f" (UID: "e43bb0c6-0f01-4106-99f8-7a2c8985315f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.208823 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.208794 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tls-certs" pod "e43bb0c6-0f01-4106-99f8-7a2c8985315f" (UID: "e43bb0c6-0f01-4106-99f8-7a2c8985315f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:37.308373 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.308344 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43bb0c6-0f01-4106-99f8-7a2c8985315f-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.308373 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.308370 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43bb0c6-0f01-4106-99f8-7a2c8985315f-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.333623 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.333544 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" podStartSLOduration=2.333519889 podStartE2EDuration="2.333519889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:37.328994428 +0000 UTC m=+2908.962388338" watchObservedRunningTime="2026-10-31 20:05:37.333519889 +0000 UTC m=+2908.966913797" Oct 31 20:05:37.435988 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.435927 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" podStartSLOduration=2.435906886 podStartE2EDuration="2.435906886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:37.434322419 +0000 UTC m=+2909.067716330" watchObservedRunningTime="2026-10-31 20:05:37.435906886 +0000 UTC m=+2909.069300797" Oct 31 20:05:37.538323 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.538290 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm"] Oct 31 20:05:37.542757 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.542724 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm"] Oct 31 20:05:37.646842 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.646811 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9"] Oct 31 20:05:37.651310 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.651275 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9"] Oct 31 20:05:37.851875 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.851830 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-2-w5-570915-kserve-5d567ffb86-p87n9" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:05:37.930049 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.930001 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/premium-w5-570915-kserve-784fc76bf4-m4jvm" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:05:37.984761 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.984725 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg"] Oct 31 20:05:37.985619 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.985595 6107 cpu_manager.go:414] "containerMap: removing container" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" Oct 31 20:05:37.985619 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.985622 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" Oct 31 20:05:37.985740 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.985674 6107 cpu_manager.go:414] "containerMap: removing container" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" Oct 31 20:05:37.985740 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.985681 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" Oct 31 20:05:37.985848 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.985804 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" containerName="main" Oct 31 20:05:37.985848 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.985821 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" containerName="main" Oct 31 20:05:37.991173 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:37.991149 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.016050 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.016023 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.016210 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.016054 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.016210 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.016171 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.016326 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.016209 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.016326 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.016228 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.016427 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.016359 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.016427 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.016389 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkdwr\" (UniqueName: \"kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-kube-api-access-lkdwr\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.016427 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.016420 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fd63df-0cfe-4329-980d-f0383e0406ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.016572 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.016491 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.024013 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.023981 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg"] Oct 31 20:05:38.117770 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.117672 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.117770 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.117751 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.118013 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.117778 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.118013 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.117982 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.118131 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.118030 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdwr\" (UniqueName: \"kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-kube-api-access-lkdwr\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.118131 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.118085 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fd63df-0cfe-4329-980d-f0383e0406ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.118260 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.118157 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.118260 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.118204 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.118260 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.118234 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.118417 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.118254 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.118417 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.118259 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.118828 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.118803 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.118977 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.118847 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fd63df-0cfe-4329-980d-f0383e0406ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.119107 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.119089 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.120779 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.120760 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.121168 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.121151 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.170908 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.170863 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.171050 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.171007 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdwr\" (UniqueName: \"kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-kube-api-access-lkdwr\") pod \"maas-default-gateway-openshift-default-58d898bd55-gb7lg\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.307367 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.307333 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:38.488683 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.488650 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9b87de-cba8-4ff6-ab19-9e1ce089feaf" path="/var/lib/kubelet/pods/0a9b87de-cba8-4ff6-ab19-9e1ce089feaf/volumes" Oct 31 20:05:38.489133 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.489119 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43bb0c6-0f01-4106-99f8-7a2c8985315f" path="/var/lib/kubelet/pods/e43bb0c6-0f01-4106-99f8-7a2c8985315f/volumes" Oct 31 20:05:38.724083 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.724051 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg"] Oct 31 20:05:38.725548 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:05:38.725509 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fd63df_0cfe_4329_980d_f0383e0406ef.slice/crio-d5de269e56ff79e0982d1a90ed2648b0838dcf3c8f5b269f5e1ed2ab2837da83 WatchSource:0}: Error finding container d5de269e56ff79e0982d1a90ed2648b0838dcf3c8f5b269f5e1ed2ab2837da83: Status 404 returned error can't find the container with id d5de269e56ff79e0982d1a90ed2648b0838dcf3c8f5b269f5e1ed2ab2837da83 Oct 31 20:05:38.728371 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.728339 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:05:38.728451 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.728403 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:05:38.728451 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:38.728433 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:05:39.045731 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:39.045608 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" event={"ID":"b6fd63df-0cfe-4329-980d-f0383e0406ef","Type":"ContainerStarted","Data":"41eb49ab14ce295242f2bbcbd90f3ea279274f6dd03c1a4347dd248e219569f8"} Oct 31 20:05:39.045731 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:39.045698 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" event={"ID":"b6fd63df-0cfe-4329-980d-f0383e0406ef","Type":"ContainerStarted","Data":"d5de269e56ff79e0982d1a90ed2648b0838dcf3c8f5b269f5e1ed2ab2837da83"} Oct 31 20:05:39.158752 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:39.158701 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" podStartSLOduration=2.158685406 podStartE2EDuration="2.158685406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:39.155926091 +0000 UTC m=+2910.789319999" watchObservedRunningTime="2026-10-31 20:05:39.158685406 +0000 UTC m=+2910.792079316" Oct 31 20:05:39.308069 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:39.307987 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:39.309733 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:39.309702 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gb7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 20:05:39.309863 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:39.309766 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 20:05:40.328907 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:40.312375 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gb7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 20:05:40.328907 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:40.312451 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 20:05:41.315908 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:41.310312 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gb7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 20:05:41.315908 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:41.310379 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 20:05:41.361316 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:41.361276 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" podUID="8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:05:42.313424 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:42.313379 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gb7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 20:05:42.313757 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:42.313722 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 20:05:43.310331 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:43.310291 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gb7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 20:05:43.310979 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:43.310948 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 20:05:43.373296 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:43.373258 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" podUID="801671a2-4d7f-4734-8a10-1d7975dda195" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:05:44.315906 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:44.311242 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gb7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 20:05:44.315906 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:44.311303 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 20:05:45.307943 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:45.307896 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gb7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 20:05:45.308156 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:45.307969 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 20:05:46.308498 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:46.308458 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gb7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 20:05:46.308869 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:46.308537 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 20:05:47.384155 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:47.384125 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:47.385059 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:47.384583 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:47.386261 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:47.386237 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:05:48.039450 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:48.039421 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" Oct 31 20:05:48.039759 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:48.039735 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" Oct 31 20:05:50.404577 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:05:50.404542 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:05:51.361060 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:51.361017 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" podUID="8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:05:53.373384 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:53.373352 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" podUID="801671a2-4d7f-4734-8a10-1d7975dda195" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:05:53.974624 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:53.974600 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" Oct 31 20:05:54.132056 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.132024 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-kube-api-access-vffr4\" (UniqueName: \"kubernetes.io/projected/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-kube-api-access-vffr4\") pod \"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35\" (UID: \"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35\") " Oct 31 20:05:54.132332 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.132071 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-plugins-config-volume\") pod \"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35\" (UID: \"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35\") " Oct 31 20:05:54.132510 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.132486 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-plugins-config-volume" pod "8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35" (UID: "8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:54.134234 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.134210 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-kube-api-access-vffr4" pod "8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35" (UID: "8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35"). InnerVolumeSpecName "kube-api-access-vffr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:54.146510 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.146482 6107 generic.go:356] "Generic (PLEG): container finished" podID="8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35" containerID="7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f" exitCode=0 Oct 31 20:05:54.146608 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.146509 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" event={"ID":"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35","Type":"ContainerDied","Data":"7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f"} Oct 31 20:05:54.146608 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.146530 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" Oct 31 20:05:54.146608 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.146558 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl" event={"ID":"8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35","Type":"ContainerDied","Data":"8cdc12eac6e5f56304daab47c6eb7121b7040647d5b37ff17f1357df447a49ac"} Oct 31 20:05:54.146608 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.146582 6107 scope.go:122] "RemoveContainer" containerID="7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f" Oct 31 20:05:54.157798 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.157774 6107 scope.go:122] "RemoveContainer" containerID="7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f" Oct 31 20:05:54.158139 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:05:54.158121 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f\": container with ID starting with 7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f not found: ID does not exist" containerID="7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f" Oct 31 20:05:54.158223 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.158155 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f"} err="failed to get container status \"7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f\": rpc error: code = NotFound desc = could not find container \"7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f\": container with ID starting with 7d21e8cdb34fd514daae72e1ad3a042ec9808e49418b4154e3f1e683b773d60f not found: ID does not exist" Oct 31 20:05:54.233638 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.233612 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vffr4\" (UniqueName: \"kubernetes.io/projected/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-kube-api-access-vffr4\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:54.233638 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.233636 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:54.298294 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.298264 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl"] Oct 31 20:05:54.302122 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.302098 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b5155c-7cd56c6999-689kl"] Oct 31 20:05:54.487518 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:54.487447 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35" path="/var/lib/kubelet/pods/8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35/volumes" Oct 31 20:05:56.606798 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:56.606720 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" Oct 31 20:05:56.774022 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:56.773987 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/801671a2-4d7f-4734-8a10-1d7975dda195-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801671a2-4d7f-4734-8a10-1d7975dda195-plugins-config-volume\") pod \"801671a2-4d7f-4734-8a10-1d7975dda195\" (UID: \"801671a2-4d7f-4734-8a10-1d7975dda195\") " Oct 31 20:05:56.774203 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:56.774079 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801671a2-4d7f-4734-8a10-1d7975dda195-kube-api-access-t7z5r\" (UniqueName: \"kubernetes.io/projected/801671a2-4d7f-4734-8a10-1d7975dda195-kube-api-access-t7z5r\") pod \"801671a2-4d7f-4734-8a10-1d7975dda195\" (UID: \"801671a2-4d7f-4734-8a10-1d7975dda195\") " Oct 31 20:05:56.774391 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:56.774367 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801671a2-4d7f-4734-8a10-1d7975dda195-plugins-config-volume" pod "801671a2-4d7f-4734-8a10-1d7975dda195" (UID: "801671a2-4d7f-4734-8a10-1d7975dda195"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:56.774548 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:56.774534 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801671a2-4d7f-4734-8a10-1d7975dda195-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:56.776127 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:56.776107 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801671a2-4d7f-4734-8a10-1d7975dda195-kube-api-access-t7z5r" pod "801671a2-4d7f-4734-8a10-1d7975dda195" (UID: "801671a2-4d7f-4734-8a10-1d7975dda195"). InnerVolumeSpecName "kube-api-access-t7z5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:56.876063 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:56.875975 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7z5r\" (UniqueName: \"kubernetes.io/projected/801671a2-4d7f-4734-8a10-1d7975dda195-kube-api-access-t7z5r\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:57.172637 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:57.172541 6107 generic.go:356] "Generic (PLEG): container finished" podID="801671a2-4d7f-4734-8a10-1d7975dda195" containerID="cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80" exitCode=0 Oct 31 20:05:57.172637 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:57.172586 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" Oct 31 20:05:57.172845 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:57.172631 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" event={"ID":"801671a2-4d7f-4734-8a10-1d7975dda195","Type":"ContainerDied","Data":"cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80"} Oct 31 20:05:57.172845 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:57.172681 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl" event={"ID":"801671a2-4d7f-4734-8a10-1d7975dda195","Type":"ContainerDied","Data":"6c5c338340a8d8d87dd753e63115d33b6510baddeeb25fcb86d008d1c6b6b49b"} Oct 31 20:05:57.172845 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:57.172704 6107 scope.go:122] "RemoveContainer" containerID="cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80" Oct 31 20:05:57.182633 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:57.182609 6107 scope.go:122] "RemoveContainer" containerID="cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80" Oct 31 20:05:57.182923 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:05:57.182904 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80\": container with ID starting with cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80 not found: ID does not exist" containerID="cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80" Oct 31 20:05:57.182982 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:57.182933 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80"} err="failed to get container status \"cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80\": rpc error: code = NotFound desc = could not find container \"cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80\": container with ID starting with cf0a71ba0ff101b22fe75b4bd13446e84a8586ee4c06412bbf9ca5107971ee80 not found: ID does not exist" Oct 31 20:05:57.326147 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:57.326113 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl"] Oct 31 20:05:57.371465 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:57.371433 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-570915-7f76b7fdf9-zfnsl"] Oct 31 20:05:58.496950 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:05:58.496912 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="801671a2-4d7f-4734-8a10-1d7975dda195" path="/var/lib/kubelet/pods/801671a2-4d7f-4734-8a10-1d7975dda195/volumes" Oct 31 20:06:20.406966 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:06:20.406926 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:06:36.254667 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:36.254625 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk"] Oct 31 20:06:36.255269 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:36.254836 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" containerID="cri-o://eabf1d7586ea8c283d953e91aa4f9b442f191bdf5997872eb1ed6a0426d6c340" gracePeriod=30 Oct 31 20:06:41.107176 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.107133 6107 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:06:41.107692 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.107206 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:06:41.465489 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.465440 6107 generic.go:356] "Generic (PLEG): container finished" podID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerID="eabf1d7586ea8c283d953e91aa4f9b442f191bdf5997872eb1ed6a0426d6c340" exitCode=0 Oct 31 20:06:41.465692 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.465670 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" event={"ID":"cf8701ec-59c6-45e1-a62f-f50d102f9758","Type":"ContainerDied","Data":"eabf1d7586ea8c283d953e91aa4f9b442f191bdf5997872eb1ed6a0426d6c340"} Oct 31 20:06:41.465754 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.465745 6107 scope.go:122] "RemoveContainer" containerID="dd10718f2efadf3cdacba77c8a8375023dfdfe6dc7d92ea70052f4f01d6faf46" Oct 31 20:06:41.521847 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.521817 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 20:06:41.669334 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669236 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-credential-socket\") pod \"cf8701ec-59c6-45e1-a62f-f50d102f9758\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " Oct 31 20:06:41.669334 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669284 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-data\") pod \"cf8701ec-59c6-45e1-a62f-f50d102f9758\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " Oct 31 20:06:41.669334 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669330 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-kube-api-access-pq27c\" (UniqueName: \"kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-kube-api-access-pq27c\") pod \"cf8701ec-59c6-45e1-a62f-f50d102f9758\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " Oct 31 20:06:41.669644 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669383 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-certs\") pod \"cf8701ec-59c6-45e1-a62f-f50d102f9758\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " Oct 31 20:06:41.669644 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669457 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf8701ec-59c6-45e1-a62f-f50d102f9758-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf8701ec-59c6-45e1-a62f-f50d102f9758-istiod-ca-cert\") pod \"cf8701ec-59c6-45e1-a62f-f50d102f9758\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " Oct 31 20:06:41.669644 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669506 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-podinfo\") pod \"cf8701ec-59c6-45e1-a62f-f50d102f9758\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " Oct 31 20:06:41.669644 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669571 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-credential-socket" pod "cf8701ec-59c6-45e1-a62f-f50d102f9758" (UID: "cf8701ec-59c6-45e1-a62f-f50d102f9758"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:41.669644 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669633 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-socket\") pod \"cf8701ec-59c6-45e1-a62f-f50d102f9758\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " Oct 31 20:06:41.669962 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669673 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-data" pod "cf8701ec-59c6-45e1-a62f-f50d102f9758" (UID: "cf8701ec-59c6-45e1-a62f-f50d102f9758"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:41.669962 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669710 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-token\" (UniqueName: \"kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-token\") pod \"cf8701ec-59c6-45e1-a62f-f50d102f9758\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " Oct 31 20:06:41.669962 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669740 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-certs" pod "cf8701ec-59c6-45e1-a62f-f50d102f9758" (UID: "cf8701ec-59c6-45e1-a62f-f50d102f9758"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:41.669962 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669788 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-envoy\") pod \"cf8701ec-59c6-45e1-a62f-f50d102f9758\" (UID: \"cf8701ec-59c6-45e1-a62f-f50d102f9758\") " Oct 31 20:06:41.669962 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669904 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-socket" pod "cf8701ec-59c6-45e1-a62f-f50d102f9758" (UID: "cf8701ec-59c6-45e1-a62f-f50d102f9758"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:41.670224 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.669964 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8701ec-59c6-45e1-a62f-f50d102f9758-istiod-ca-cert" pod "cf8701ec-59c6-45e1-a62f-f50d102f9758" (UID: "cf8701ec-59c6-45e1-a62f-f50d102f9758"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:06:41.670311 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.670286 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:41.670311 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.670308 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:41.670432 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.670318 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:41.670432 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.670328 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:41.670432 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.670336 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf8701ec-59c6-45e1-a62f-f50d102f9758-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:41.671775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.671751 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-podinfo" pod "cf8701ec-59c6-45e1-a62f-f50d102f9758" (UID: "cf8701ec-59c6-45e1-a62f-f50d102f9758"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:06:41.671955 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.671841 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-envoy" pod "cf8701ec-59c6-45e1-a62f-f50d102f9758" (UID: "cf8701ec-59c6-45e1-a62f-f50d102f9758"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:41.672138 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.672120 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-kube-api-access-pq27c" pod "cf8701ec-59c6-45e1-a62f-f50d102f9758" (UID: "cf8701ec-59c6-45e1-a62f-f50d102f9758"). InnerVolumeSpecName "kube-api-access-pq27c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:41.672185 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.672141 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-token" pod "cf8701ec-59c6-45e1-a62f-f50d102f9758" (UID: "cf8701ec-59c6-45e1-a62f-f50d102f9758"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:41.771113 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.771082 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:41.771113 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.771111 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:41.771113 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.771127 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq27c\" (UniqueName: \"kubernetes.io/projected/cf8701ec-59c6-45e1-a62f-f50d102f9758-kube-api-access-pq27c\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:41.771369 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:41.771140 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf8701ec-59c6-45e1-a62f-f50d102f9758-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:42.473120 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:42.473087 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" Oct 31 20:06:42.473638 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:42.473121 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk" event={"ID":"cf8701ec-59c6-45e1-a62f-f50d102f9758","Type":"ContainerDied","Data":"f5bc2f55a18f73f9b6d05ef6aa1de69c544d589bfd0dd7b4b5f9ffe9ce29a5b3"} Oct 31 20:06:42.473638 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:42.473181 6107 scope.go:122] "RemoveContainer" containerID="eabf1d7586ea8c283d953e91aa4f9b442f191bdf5997872eb1ed6a0426d6c340" Oct 31 20:06:42.671255 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:42.671224 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk"] Oct 31 20:06:42.717001 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:42.716968 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-bn9xk"] Oct 31 20:06:44.081167 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:44.081128 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4"] Oct 31 20:06:44.185262 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:44.185228 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5"] Oct 31 20:06:44.487403 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:44.487369 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" path="/var/lib/kubelet/pods/cf8701ec-59c6-45e1-a62f-f50d102f9758/volumes" Oct 31 20:06:49.088251 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:49.087875 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" podUID="d12965bb-f23f-41cf-b6cc-592e86d1d8f7" containerName="payload-processing" containerID="cri-o://280664fd9a82876e8b64e7f9fa62555c38f59f4e0010646877af36779ba4df1f" gracePeriod=25 Oct 31 20:06:49.185775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:49.185704 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" podUID="b5f42617-d805-4421-9110-f23649003317" containerName="payload-pre-processing" containerID="cri-o://530400e396256edf95b9ee1a803f1f38100883a5135f918d97fe0e7d730607f2" gracePeriod=25 Oct 31 20:06:49.350258 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:49.350173 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" podUID="b5f42617-d805-4421-9110-f23649003317" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:06:49.350258 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:49.350198 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" podUID="d12965bb-f23f-41cf-b6cc-592e86d1d8f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:06:50.404409 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:06:50.404361 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:06:59.349575 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:59.349515 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" podUID="d12965bb-f23f-41cf-b6cc-592e86d1d8f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:06:59.350079 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:06:59.349628 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" podUID="b5f42617-d805-4421-9110-f23649003317" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:07:09.348876 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:07:09.348794 6107 logging.go:55] [core] [Channel #6212 SubChannel #6214]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.43:50052", ServerName: "10.133.0.43:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.43:50052: connect: connection refused" Oct 31 20:07:09.348876 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:07:09.348794 6107 logging.go:55] [core] [Channel #6213 SubChannel #6215]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.44:50052", ServerName: "10.133.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.44:50052: connect: connection refused" Oct 31 20:07:09.459503 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.459480 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" Oct 31 20:07:09.488896 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.488864 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" Oct 31 20:07:09.508951 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.508923 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-plugins-config-volume\") pod \"d12965bb-f23f-41cf-b6cc-592e86d1d8f7\" (UID: \"d12965bb-f23f-41cf-b6cc-592e86d1d8f7\") " Oct 31 20:07:09.509087 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.509008 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f42617-d805-4421-9110-f23649003317-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f42617-d805-4421-9110-f23649003317-plugins-config-volume\") pod \"b5f42617-d805-4421-9110-f23649003317\" (UID: \"b5f42617-d805-4421-9110-f23649003317\") " Oct 31 20:07:09.509087 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.509062 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f42617-d805-4421-9110-f23649003317-kube-api-access-d4wgn\" (UniqueName: \"kubernetes.io/projected/b5f42617-d805-4421-9110-f23649003317-kube-api-access-d4wgn\") pod \"b5f42617-d805-4421-9110-f23649003317\" (UID: \"b5f42617-d805-4421-9110-f23649003317\") " Oct 31 20:07:09.509163 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.509127 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-kube-api-access-v8d8d\" (UniqueName: \"kubernetes.io/projected/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-kube-api-access-v8d8d\") pod \"d12965bb-f23f-41cf-b6cc-592e86d1d8f7\" (UID: \"d12965bb-f23f-41cf-b6cc-592e86d1d8f7\") " Oct 31 20:07:09.509388 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.509364 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-plugins-config-volume" pod "d12965bb-f23f-41cf-b6cc-592e86d1d8f7" (UID: "d12965bb-f23f-41cf-b6cc-592e86d1d8f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:07:09.509435 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.509414 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f42617-d805-4421-9110-f23649003317-plugins-config-volume" pod "b5f42617-d805-4421-9110-f23649003317" (UID: "b5f42617-d805-4421-9110-f23649003317"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:07:09.511132 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.511107 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f42617-d805-4421-9110-f23649003317-kube-api-access-d4wgn" pod "b5f42617-d805-4421-9110-f23649003317" (UID: "b5f42617-d805-4421-9110-f23649003317"). InnerVolumeSpecName "kube-api-access-d4wgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:07:09.511240 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.511218 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-kube-api-access-v8d8d" pod "d12965bb-f23f-41cf-b6cc-592e86d1d8f7" (UID: "d12965bb-f23f-41cf-b6cc-592e86d1d8f7"). InnerVolumeSpecName "kube-api-access-v8d8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:07:09.610973 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.610867 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:09.610973 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.610920 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f42617-d805-4421-9110-f23649003317-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:09.610973 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.610936 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4wgn\" (UniqueName: \"kubernetes.io/projected/b5f42617-d805-4421-9110-f23649003317-kube-api-access-d4wgn\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:09.610973 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.610949 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8d8d\" (UniqueName: \"kubernetes.io/projected/d12965bb-f23f-41cf-b6cc-592e86d1d8f7-kube-api-access-v8d8d\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:09.622656 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.622621 6107 scope.go:122] "RemoveContainer" containerID="54490b04ccc0037a9ca99376db460669d10669fbbf8a4ae85e9b5111354a1a69" Oct 31 20:07:09.632726 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.632701 6107 scope.go:122] "RemoveContainer" containerID="280664fd9a82876e8b64e7f9fa62555c38f59f4e0010646877af36779ba4df1f" Oct 31 20:07:09.641192 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.641163 6107 scope.go:122] "RemoveContainer" containerID="530400e396256edf95b9ee1a803f1f38100883a5135f918d97fe0e7d730607f2" Oct 31 20:07:09.712239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.712198 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" event={"ID":"b5f42617-d805-4421-9110-f23649003317","Type":"ContainerDied","Data":"530400e396256edf95b9ee1a803f1f38100883a5135f918d97fe0e7d730607f2"} Oct 31 20:07:09.712434 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.712253 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" Oct 31 20:07:09.712434 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.712273 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" event={"ID":"b5f42617-d805-4421-9110-f23649003317","Type":"ContainerDied","Data":"82732f7dbc1f732048df7175151cf72356673f229a00c13eeda34561a81d1d0c"} Oct 31 20:07:09.712434 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.712300 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" event={"ID":"d12965bb-f23f-41cf-b6cc-592e86d1d8f7","Type":"ContainerDied","Data":"280664fd9a82876e8b64e7f9fa62555c38f59f4e0010646877af36779ba4df1f"} Oct 31 20:07:09.712434 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.712319 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" event={"ID":"d12965bb-f23f-41cf-b6cc-592e86d1d8f7","Type":"ContainerDied","Data":"e69aaadddd0befc71e240a647904adf5d8b15410c593477e6157c01a81e6eb87"} Oct 31 20:07:09.712434 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:09.712254 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" Oct 31 20:07:10.032280 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:10.032244 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4"] Oct 31 20:07:10.039114 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:10.039088 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4"] Oct 31 20:07:10.138786 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:10.138746 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5"] Oct 31 20:07:10.142973 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:10.142949 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5"] Oct 31 20:07:10.349201 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:10.349112 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-5a4438-785f9cf4d5-tt9q4" podUID="d12965bb-f23f-41cf-b6cc-592e86d1d8f7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.44:50052\" within 1s: context deadline exceeded" Oct 31 20:07:10.349612 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:10.349105 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5a4438-754859bc47-btlg5" podUID="b5f42617-d805-4421-9110-f23649003317" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.43:50052\" within 1s: context deadline exceeded" Oct 31 20:07:10.349612 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:07:10.349270 6107 logging.go:55] [core] [Channel #6213 SubChannel #6215]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.44:50052", ServerName: "10.133.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.44:50052: operation was canceled" Oct 31 20:07:10.486598 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:10.486554 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f42617-d805-4421-9110-f23649003317" path="/var/lib/kubelet/pods/b5f42617-d805-4421-9110-f23649003317/volumes" Oct 31 20:07:10.487100 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:10.487079 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12965bb-f23f-41cf-b6cc-592e86d1d8f7" path="/var/lib/kubelet/pods/d12965bb-f23f-41cf-b6cc-592e86d1d8f7/volumes" Oct 31 20:07:19.286580 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.286540 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9"] Oct 31 20:07:19.287261 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.286746 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" podUID="73a00f61-6ddd-462c-a08a-7bd2eb1b226e" containerName="discovery" containerID="cri-o://8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a" gracePeriod=30 Oct 31 20:07:19.557723 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.557695 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 20:07:19.642345 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.642311 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-kubeconfig\") pod \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " Oct 31 20:07:19.642546 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.642493 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-local-certs\") pod \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " Oct 31 20:07:19.642618 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.642551 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-kube-api-access-c8b9x\" (UniqueName: \"kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-kube-api-access-c8b9x\") pod \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " Oct 31 20:07:19.642618 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.642591 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-cacerts\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-cacerts\") pod \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " Oct 31 20:07:19.642728 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.642665 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-token\" (UniqueName: \"kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-token\") pod \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " Oct 31 20:07:19.642779 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.642766 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-ca-configmap\") pod \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " Oct 31 20:07:19.642840 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.642825 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-dns-cert\") pod \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\" (UID: \"73a00f61-6ddd-462c-a08a-7bd2eb1b226e\") " Oct 31 20:07:19.643455 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.643417 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-ca-configmap" pod "73a00f61-6ddd-462c-a08a-7bd2eb1b226e" (UID: "73a00f61-6ddd-462c-a08a-7bd2eb1b226e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:07:19.645275 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.645241 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-kubeconfig" pod "73a00f61-6ddd-462c-a08a-7bd2eb1b226e" (UID: "73a00f61-6ddd-462c-a08a-7bd2eb1b226e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:07:19.645475 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.645450 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-cacerts" pod "73a00f61-6ddd-462c-a08a-7bd2eb1b226e" (UID: "73a00f61-6ddd-462c-a08a-7bd2eb1b226e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:07:19.645560 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.645503 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-kube-api-access-c8b9x" pod "73a00f61-6ddd-462c-a08a-7bd2eb1b226e" (UID: "73a00f61-6ddd-462c-a08a-7bd2eb1b226e"). InnerVolumeSpecName "kube-api-access-c8b9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:07:19.645604 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.645586 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-token" pod "73a00f61-6ddd-462c-a08a-7bd2eb1b226e" (UID: "73a00f61-6ddd-462c-a08a-7bd2eb1b226e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:07:19.645698 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.645684 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-local-certs" pod "73a00f61-6ddd-462c-a08a-7bd2eb1b226e" (UID: "73a00f61-6ddd-462c-a08a-7bd2eb1b226e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:19.645743 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.645692 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-dns-cert" pod "73a00f61-6ddd-462c-a08a-7bd2eb1b226e" (UID: "73a00f61-6ddd-462c-a08a-7bd2eb1b226e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:07:19.744786 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.744756 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:19.744786 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.744784 6107 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-ca-configmap\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:19.744786 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.744794 6107 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-csr-dns-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:19.745069 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.744804 6107 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-istio-kubeconfig\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:19.745069 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.744816 6107 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-local-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:19.745069 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.744824 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8b9x\" (UniqueName: \"kubernetes.io/projected/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-kube-api-access-c8b9x\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:19.745069 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.744832 6107 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73a00f61-6ddd-462c-a08a-7bd2eb1b226e-cacerts\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:19.777997 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.777905 6107 generic.go:356] "Generic (PLEG): container finished" podID="73a00f61-6ddd-462c-a08a-7bd2eb1b226e" containerID="8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a" exitCode=0 Oct 31 20:07:19.778417 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.778089 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" event={"ID":"73a00f61-6ddd-462c-a08a-7bd2eb1b226e","Type":"ContainerDied","Data":"8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a"} Oct 31 20:07:19.778417 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.778134 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" event={"ID":"73a00f61-6ddd-462c-a08a-7bd2eb1b226e","Type":"ContainerDied","Data":"59b3f8b46b6d2568a799d92feb3b1a6b0786319bfb0705684226d1286899d3ef"} Oct 31 20:07:19.778417 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.778158 6107 scope.go:122] "RemoveContainer" containerID="8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a" Oct 31 20:07:19.778417 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.778324 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9" Oct 31 20:07:19.789548 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.789512 6107 scope.go:122] "RemoveContainer" containerID="8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a" Oct 31 20:07:19.789914 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:07:19.789871 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a\": container with ID starting with 8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a not found: ID does not exist" containerID="8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a" Oct 31 20:07:19.790006 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.789924 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a"} err="failed to get container status \"8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a\": rpc error: code = NotFound desc = could not find container \"8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a\": container with ID starting with 8588c50ec9387bf7f07943bb051566f794bc2bc5268d4b080f978730577bb32a not found: ID does not exist" Oct 31 20:07:19.935722 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.935679 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9"] Oct 31 20:07:19.939015 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:19.938984 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7sfw9"] Oct 31 20:07:20.404296 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:07:20.404260 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:07:20.486408 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:20.486369 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a00f61-6ddd-462c-a08a-7bd2eb1b226e" path="/var/lib/kubelet/pods/73a00f61-6ddd-462c-a08a-7bd2eb1b226e/volumes" Oct 31 20:07:41.799121 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.799084 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4"] Oct 31 20:07:41.799900 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.799856 6107 cpu_manager.go:414] "containerMap: removing container" podUID="b5f42617-d805-4421-9110-f23649003317" containerName="payload-pre-processing" Oct 31 20:07:41.799900 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.799899 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f42617-d805-4421-9110-f23649003317" containerName="payload-pre-processing" Oct 31 20:07:41.800054 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.799924 6107 cpu_manager.go:414] "containerMap: removing container" podUID="73a00f61-6ddd-462c-a08a-7bd2eb1b226e" containerName="discovery" Oct 31 20:07:41.800054 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.799933 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a00f61-6ddd-462c-a08a-7bd2eb1b226e" containerName="discovery" Oct 31 20:07:41.800054 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.799950 6107 cpu_manager.go:414] "containerMap: removing container" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" Oct 31 20:07:41.800054 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.799957 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" Oct 31 20:07:41.800054 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.799981 6107 cpu_manager.go:414] "containerMap: removing container" podUID="801671a2-4d7f-4734-8a10-1d7975dda195" containerName="payload-pre-processing" Oct 31 20:07:41.800054 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.799989 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801671a2-4d7f-4734-8a10-1d7975dda195" containerName="payload-pre-processing" Oct 31 20:07:41.800054 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800007 6107 cpu_manager.go:414] "containerMap: removing container" podUID="d12965bb-f23f-41cf-b6cc-592e86d1d8f7" containerName="payload-processing" Oct 31 20:07:41.800054 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800014 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12965bb-f23f-41cf-b6cc-592e86d1d8f7" containerName="payload-processing" Oct 31 20:07:41.800054 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800026 6107 cpu_manager.go:414] "containerMap: removing container" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" Oct 31 20:07:41.800054 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800034 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" Oct 31 20:07:41.800389 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800060 6107 cpu_manager.go:414] "containerMap: removing container" podUID="8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35" containerName="payload-processing" Oct 31 20:07:41.800389 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800067 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35" containerName="payload-processing" Oct 31 20:07:41.800389 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800211 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f42617-d805-4421-9110-f23649003317" containerName="payload-pre-processing" Oct 31 20:07:41.800389 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800224 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a00f61-6ddd-462c-a08a-7bd2eb1b226e" containerName="discovery" Oct 31 20:07:41.800389 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800236 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" Oct 31 20:07:41.800389 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800248 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="801671a2-4d7f-4734-8a10-1d7975dda195" containerName="payload-pre-processing" Oct 31 20:07:41.800389 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800258 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12965bb-f23f-41cf-b6cc-592e86d1d8f7" containerName="payload-processing" Oct 31 20:07:41.800389 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800267 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8701ec-59c6-45e1-a62f-f50d102f9758" containerName="istio-proxy" Oct 31 20:07:41.800389 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.800277 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5ade3d-0e5f-429d-a49c-2c78ed7c1f35" containerName="payload-processing" Oct 31 20:07:41.804259 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.804234 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:41.863557 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.863527 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4"] Oct 31 20:07:41.906193 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.906153 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a89a58\"/\"openshift-service-ca.crt\"" Oct 31 20:07:41.906193 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.906181 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a89a58\"/\"kube-root-ca.crt\"" Oct 31 20:07:41.906604 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.906532 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a89a58\"/\"test-model-a89a58-kserve-self-signed-certs\"" Oct 31 20:07:41.906715 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.906665 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a89a58\"/\"default-dockercfg-xlw6l\"" Oct 31 20:07:41.916277 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.908967 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-home\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:41.916277 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.909049 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-dshm\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:41.916277 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.909153 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgvq8\" (UniqueName: \"kubernetes.io/projected/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-kube-api-access-zgvq8\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:41.916277 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.909233 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tls-certs\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:41.916277 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.909305 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tmp-dir\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:41.916277 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:41.909333 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-model-cache\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.010401 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.010370 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-home\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.010581 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.010425 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-dshm\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.010581 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.010467 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgvq8\" (UniqueName: \"kubernetes.io/projected/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-kube-api-access-zgvq8\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.010581 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.010517 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tls-certs\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.010765 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.010579 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tmp-dir\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.010765 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.010652 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-model-cache\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.010765 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.010740 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-home\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.011004 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.010938 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tmp-dir\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.011050 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.011021 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-model-cache\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.012689 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.012660 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-dshm\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.013015 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.012996 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tls-certs\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.063596 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.063526 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgvq8\" (UniqueName: \"kubernetes.io/projected/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-kube-api-access-zgvq8\") pod \"test-model-a89a58-kserve-78bcc66955-7dfr4\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.116959 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.116930 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:42.507349 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.507315 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4"] Oct 31 20:07:42.507920 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:07:42.507866 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c8bf88c_41d0_4e8b_9f0c_5c9e6966f217.slice/crio-d996683f23cff6ec8280ccb9d04bff8341c8bd8de77f899bd4f96c14058d55a4 WatchSource:0}: Error finding container d996683f23cff6ec8280ccb9d04bff8341c8bd8de77f899bd4f96c14058d55a4: Status 404 returned error can't find the container with id d996683f23cff6ec8280ccb9d04bff8341c8bd8de77f899bd4f96c14058d55a4 Oct 31 20:07:42.934290 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.934202 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" event={"ID":"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217","Type":"ContainerStarted","Data":"af5eaa5a040246e813717af210ee13cd767a2a44e768a7d3896a7eb827965328"} Oct 31 20:07:42.934290 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:42.934246 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" event={"ID":"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217","Type":"ContainerStarted","Data":"d996683f23cff6ec8280ccb9d04bff8341c8bd8de77f899bd4f96c14058d55a4"} Oct 31 20:07:43.041963 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:43.041916 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" podStartSLOduration=2.04190259 podStartE2EDuration="2.04190259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:07:43.040728672 +0000 UTC m=+3034.674122593" watchObservedRunningTime="2026-10-31 20:07:43.04190259 +0000 UTC m=+3034.675296492" Oct 31 20:07:43.117770 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:43.117736 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:50.403531 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:07:50.403489 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:07:52.118239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:52.118024 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:52.132316 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:52.132284 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:07:53.006348 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:07:53.006318 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:08:05.034133 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:05.034098 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r"] Oct 31 20:08:05.034646 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:05.034290 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" podUID="ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" containerName="istio-proxy" containerID="cri-o://d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58" gracePeriod=30 Oct 31 20:08:07.710335 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:07.710299 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6"] Oct 31 20:08:10.297733 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.297709 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 20:08:10.354063 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.353987 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-data\") pod \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " Oct 31 20:08:10.354063 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354040 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-socket\") pod \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " Oct 31 20:08:10.354250 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354076 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-kube-api-access-fbmh9\" (UniqueName: \"kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-kube-api-access-fbmh9\") pod \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " Oct 31 20:08:10.354250 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354163 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-credential-socket\") pod \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " Oct 31 20:08:10.354250 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354245 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-certs\") pod \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " Oct 31 20:08:10.354403 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354272 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-envoy\") pod \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " Oct 31 20:08:10.354403 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354274 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-data" pod "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" (UID: "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:10.354403 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354290 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-socket" pod "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" (UID: "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:10.354403 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354301 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-token\" (UniqueName: \"kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-token\") pod \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " Oct 31 20:08:10.354403 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354352 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-podinfo\") pod \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " Oct 31 20:08:10.354403 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354400 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istiod-ca-cert\") pod \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\" (UID: \"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79\") " Oct 31 20:08:10.354691 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354548 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-certs" pod "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" (UID: "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:10.354691 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354565 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-credential-socket" pod "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" (UID: "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:10.354928 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354875 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istiod-ca-cert" pod "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" (UID: "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:08:10.355017 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354925 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:10.355017 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354948 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:10.355017 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354964 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:10.355017 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.354977 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:10.356412 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.356390 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-envoy" pod "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" (UID: "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:10.356517 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.356415 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-token" pod "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" (UID: "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:10.356517 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.356477 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-podinfo" pod "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" (UID: "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:08:10.356517 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.356498 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-kube-api-access-fbmh9" pod "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" (UID: "ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79"). InnerVolumeSpecName "kube-api-access-fbmh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:10.455895 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.455856 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:10.456088 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.455917 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:10.456088 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.455936 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbmh9\" (UniqueName: \"kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-kube-api-access-fbmh9\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:10.456088 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.455946 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:10.456088 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:10.455956 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:11.114331 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:11.114299 6107 generic.go:356] "Generic (PLEG): container finished" podID="ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" containerID="d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58" exitCode=0 Oct 31 20:08:11.114491 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:11.114362 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" Oct 31 20:08:11.114491 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:11.114397 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" event={"ID":"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79","Type":"ContainerDied","Data":"d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58"} Oct 31 20:08:11.114491 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:11.114475 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r" event={"ID":"ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79","Type":"ContainerDied","Data":"43c30ed80d5f12aa8e7c001bc12d1df7b718222b61c5a271b4aeeb64bc60944d"} Oct 31 20:08:11.114616 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:11.114509 6107 scope.go:122] "RemoveContainer" containerID="d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58" Oct 31 20:08:11.126475 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:11.126454 6107 scope.go:122] "RemoveContainer" containerID="d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58" Oct 31 20:08:11.126746 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:08:11.126728 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58\": container with ID starting with d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58 not found: ID does not exist" containerID="d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58" Oct 31 20:08:11.126828 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:11.126753 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58"} err="failed to get container status \"d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58\": rpc error: code = NotFound desc = could not find container \"d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58\": container with ID starting with d81e78931b3594206476db15311f68db9139d7faced67d99d152540253df2b58 not found: ID does not exist" Oct 31 20:08:11.318865 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:11.318832 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r"] Oct 31 20:08:11.324735 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:11.324709 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-wll2r"] Oct 31 20:08:12.486333 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:12.486301 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" path="/var/lib/kubelet/pods/ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79/volumes" Oct 31 20:08:20.405136 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:08:20.405057 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:08:22.732218 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:22.732141 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" containerID="cri-o://0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e" gracePeriod=45 Oct 31 20:08:23.000576 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.000548 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 20:08:23.037500 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.037455 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f638482-f128-45a1-8a74-3d9751414c81-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f638482-f128-45a1-8a74-3d9751414c81-tls-certs\") pod \"5f638482-f128-45a1-8a74-3d9751414c81\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " Oct 31 20:08:23.037702 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.037527 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-home\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-home\") pod \"5f638482-f128-45a1-8a74-3d9751414c81\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " Oct 31 20:08:23.037702 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.037558 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-tmp-dir\") pod \"5f638482-f128-45a1-8a74-3d9751414c81\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " Oct 31 20:08:23.037702 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.037580 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-dshm\") pod \"5f638482-f128-45a1-8a74-3d9751414c81\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " Oct 31 20:08:23.037702 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.037608 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-model-cache\") pod \"5f638482-f128-45a1-8a74-3d9751414c81\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " Oct 31 20:08:23.037963 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.037745 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f638482-f128-45a1-8a74-3d9751414c81-kube-api-access-62pxl\" (UniqueName: \"kubernetes.io/projected/5f638482-f128-45a1-8a74-3d9751414c81-kube-api-access-62pxl\") pod \"5f638482-f128-45a1-8a74-3d9751414c81\" (UID: \"5f638482-f128-45a1-8a74-3d9751414c81\") " Oct 31 20:08:23.038026 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.037965 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-tmp-dir" pod "5f638482-f128-45a1-8a74-3d9751414c81" (UID: "5f638482-f128-45a1-8a74-3d9751414c81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:23.038026 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.037983 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-model-cache" pod "5f638482-f128-45a1-8a74-3d9751414c81" (UID: "5f638482-f128-45a1-8a74-3d9751414c81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:23.038169 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.038146 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-home" pod "5f638482-f128-45a1-8a74-3d9751414c81" (UID: "5f638482-f128-45a1-8a74-3d9751414c81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:23.038339 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.038319 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:23.038409 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.038342 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:23.038409 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.038352 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:23.040030 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.040001 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-dshm" pod "5f638482-f128-45a1-8a74-3d9751414c81" (UID: "5f638482-f128-45a1-8a74-3d9751414c81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:23.040146 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.040091 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f638482-f128-45a1-8a74-3d9751414c81-kube-api-access-62pxl" pod "5f638482-f128-45a1-8a74-3d9751414c81" (UID: "5f638482-f128-45a1-8a74-3d9751414c81"). InnerVolumeSpecName "kube-api-access-62pxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:23.040146 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.040129 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f638482-f128-45a1-8a74-3d9751414c81-tls-certs" pod "5f638482-f128-45a1-8a74-3d9751414c81" (UID: "5f638482-f128-45a1-8a74-3d9751414c81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:08:23.139553 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.139522 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f638482-f128-45a1-8a74-3d9751414c81-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:23.139553 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.139551 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f638482-f128-45a1-8a74-3d9751414c81-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:23.139813 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.139564 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62pxl\" (UniqueName: \"kubernetes.io/projected/5f638482-f128-45a1-8a74-3d9751414c81-kube-api-access-62pxl\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:23.200272 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.200236 6107 generic.go:356] "Generic (PLEG): container finished" podID="5f638482-f128-45a1-8a74-3d9751414c81" containerID="0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e" exitCode=0 Oct 31 20:08:23.200577 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.200304 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" event={"ID":"5f638482-f128-45a1-8a74-3d9751414c81","Type":"ContainerDied","Data":"0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e"} Oct 31 20:08:23.200577 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.200352 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" Oct 31 20:08:23.200577 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.200369 6107 scope.go:122] "RemoveContainer" containerID="0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e" Oct 31 20:08:23.200577 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.200356 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6" event={"ID":"5f638482-f128-45a1-8a74-3d9751414c81","Type":"ContainerDied","Data":"5e0e2a64ade952cb5dfe338dc3dfe2326730965c95df61f7ad9003f4649ac15c"} Oct 31 20:08:23.211389 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.211371 6107 scope.go:122] "RemoveContainer" containerID="0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e" Oct 31 20:08:23.211671 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:08:23.211648 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e\": container with ID starting with 0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e not found: ID does not exist" containerID="0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e" Oct 31 20:08:23.211761 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.211684 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e"} err="failed to get container status \"0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e\": rpc error: code = NotFound desc = could not find container \"0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e\": container with ID starting with 0f2f528a9f891b2628b60bd999841bd2a01012bb1e0158906cc0813767c0097e not found: ID does not exist" Oct 31 20:08:23.358477 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.358400 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6"] Oct 31 20:08:23.363917 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:23.363892 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/premium-w3-662be1-kserve-5b9b9bc86f-dmrl6"] Oct 31 20:08:24.487285 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:24.487255 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f638482-f128-45a1-8a74-3d9751414c81" path="/var/lib/kubelet/pods/5f638482-f128-45a1-8a74-3d9751414c81/volumes" Oct 31 20:08:44.095773 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:44.095734 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl"] Oct 31 20:08:44.096324 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:44.096075 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" containerID="cri-o://792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5" gracePeriod=30 Oct 31 20:08:49.372487 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.372421 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 20:08:49.392949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.392919 6107 generic.go:356] "Generic (PLEG): container finished" podID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerID="792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5" exitCode=0 Oct 31 20:08:49.393107 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.392987 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" Oct 31 20:08:49.393107 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.393002 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" event={"ID":"704ab7da-672d-4ed0-8efa-7177dde795b0","Type":"ContainerDied","Data":"792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5"} Oct 31 20:08:49.393107 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.393059 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl" event={"ID":"704ab7da-672d-4ed0-8efa-7177dde795b0","Type":"ContainerDied","Data":"789f91bea7cd886825db00bdd8b549f8dab6c27a8b727740093d8718acd55085"} Oct 31 20:08:49.393107 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.393084 6107 scope.go:122] "RemoveContainer" containerID="792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5" Oct 31 20:08:49.404627 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.404610 6107 scope.go:122] "RemoveContainer" containerID="0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059" Oct 31 20:08:49.413606 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.413592 6107 scope.go:122] "RemoveContainer" containerID="792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5" Oct 31 20:08:49.413914 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:08:49.413857 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5\": container with ID starting with 792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5 not found: ID does not exist" containerID="792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5" Oct 31 20:08:49.413985 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.413923 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5"} err="failed to get container status \"792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5\": rpc error: code = NotFound desc = could not find container \"792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5\": container with ID starting with 792a491037455af291080b61dd996d17c7f3bd5e59b9ddb542a1f9a3a9f3e7c5 not found: ID does not exist" Oct 31 20:08:49.413985 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.413942 6107 scope.go:122] "RemoveContainer" containerID="0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059" Oct 31 20:08:49.414171 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:08:49.414152 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059\": container with ID starting with 0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059 not found: ID does not exist" containerID="0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059" Oct 31 20:08:49.414229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.414177 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059"} err="failed to get container status \"0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059\": rpc error: code = NotFound desc = could not find container \"0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059\": container with ID starting with 0b387f8de75a4a0a2965dc4143cdd167dbe4ffb13b49a561d9a2f92145f9b059 not found: ID does not exist" Oct 31 20:08:49.537277 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537244 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-data\") pod \"704ab7da-672d-4ed0-8efa-7177dde795b0\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " Oct 31 20:08:49.537277 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537281 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-socket\") pod \"704ab7da-672d-4ed0-8efa-7177dde795b0\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " Oct 31 20:08:49.537500 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537315 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-certs\") pod \"704ab7da-672d-4ed0-8efa-7177dde795b0\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " Oct 31 20:08:49.537500 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537401 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-credential-socket\") pod \"704ab7da-672d-4ed0-8efa-7177dde795b0\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " Oct 31 20:08:49.537500 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537434 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-kube-api-access-t9mg7\" (UniqueName: \"kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-kube-api-access-t9mg7\") pod \"704ab7da-672d-4ed0-8efa-7177dde795b0\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " Oct 31 20:08:49.537500 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537469 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-podinfo\") pod \"704ab7da-672d-4ed0-8efa-7177dde795b0\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " Oct 31 20:08:49.537696 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537569 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-token\" (UniqueName: \"kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-token\") pod \"704ab7da-672d-4ed0-8efa-7177dde795b0\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " Oct 31 20:08:49.537696 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537618 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-envoy\") pod \"704ab7da-672d-4ed0-8efa-7177dde795b0\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " Oct 31 20:08:49.537696 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537564 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-socket" pod "704ab7da-672d-4ed0-8efa-7177dde795b0" (UID: "704ab7da-672d-4ed0-8efa-7177dde795b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:49.537696 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537578 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-data" pod "704ab7da-672d-4ed0-8efa-7177dde795b0" (UID: "704ab7da-672d-4ed0-8efa-7177dde795b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:49.537874 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537709 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704ab7da-672d-4ed0-8efa-7177dde795b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704ab7da-672d-4ed0-8efa-7177dde795b0-istiod-ca-cert\") pod \"704ab7da-672d-4ed0-8efa-7177dde795b0\" (UID: \"704ab7da-672d-4ed0-8efa-7177dde795b0\") " Oct 31 20:08:49.537874 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537689 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-certs" pod "704ab7da-672d-4ed0-8efa-7177dde795b0" (UID: "704ab7da-672d-4ed0-8efa-7177dde795b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:49.537874 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.537753 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-credential-socket" pod "704ab7da-672d-4ed0-8efa-7177dde795b0" (UID: "704ab7da-672d-4ed0-8efa-7177dde795b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:49.538118 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.538084 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704ab7da-672d-4ed0-8efa-7177dde795b0-istiod-ca-cert" pod "704ab7da-672d-4ed0-8efa-7177dde795b0" (UID: "704ab7da-672d-4ed0-8efa-7177dde795b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:08:49.538249 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.538194 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:49.538249 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.538215 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:49.538249 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.538231 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:49.538249 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.538244 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:49.538249 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.538256 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704ab7da-672d-4ed0-8efa-7177dde795b0-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:49.539765 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.539744 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-token" pod "704ab7da-672d-4ed0-8efa-7177dde795b0" (UID: "704ab7da-672d-4ed0-8efa-7177dde795b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:49.540245 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.540221 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-envoy" pod "704ab7da-672d-4ed0-8efa-7177dde795b0" (UID: "704ab7da-672d-4ed0-8efa-7177dde795b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:49.540363 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.540343 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-kube-api-access-t9mg7" pod "704ab7da-672d-4ed0-8efa-7177dde795b0" (UID: "704ab7da-672d-4ed0-8efa-7177dde795b0"). InnerVolumeSpecName "kube-api-access-t9mg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:49.540363 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.540349 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-podinfo" pod "704ab7da-672d-4ed0-8efa-7177dde795b0" (UID: "704ab7da-672d-4ed0-8efa-7177dde795b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:08:49.639510 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.639437 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:49.639510 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.639460 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:49.639510 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.639471 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9mg7\" (UniqueName: \"kubernetes.io/projected/704ab7da-672d-4ed0-8efa-7177dde795b0-kube-api-access-t9mg7\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:49.639510 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.639481 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704ab7da-672d-4ed0-8efa-7177dde795b0-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:49.810379 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.810348 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl"] Oct 31 20:08:49.813582 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:49.813554 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-jk5hl"] Oct 31 20:08:50.403865 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:08:50.403831 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:08:50.486453 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:08:50.486410 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" path="/var/lib/kubelet/pods/704ab7da-672d-4ed0-8efa-7177dde795b0/volumes" Oct 31 20:09:20.402755 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:09:20.402715 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:09:50.403306 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:09:50.403212 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:10:20.404043 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:10:20.404000 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:10:50.404440 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:10:50.404402 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:11:20.403217 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:11:20.403129 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:11:29.243806 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:29.243774 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4"] Oct 31 20:11:44.267864 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.267820 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" podUID="0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217" containerName="main" containerID="cri-o://af5eaa5a040246e813717af210ee13cd767a2a44e768a7d3896a7eb827965328" gracePeriod=45 Oct 31 20:11:44.448492 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.448459 6107 generic.go:356] "Generic (PLEG): container finished" podID="0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217" containerID="af5eaa5a040246e813717af210ee13cd767a2a44e768a7d3896a7eb827965328" exitCode=0 Oct 31 20:11:44.448644 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.448534 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" event={"ID":"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217","Type":"ContainerDied","Data":"af5eaa5a040246e813717af210ee13cd767a2a44e768a7d3896a7eb827965328"} Oct 31 20:11:44.539599 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.539577 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:11:44.628209 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.628170 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tls-certs\") pod \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " Oct 31 20:11:44.628397 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.628249 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-dshm\") pod \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " Oct 31 20:11:44.628397 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.628317 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-home\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-home\") pod \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " Oct 31 20:11:44.628496 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.628447 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-model-cache\") pod \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " Oct 31 20:11:44.628543 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.628503 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tmp-dir\") pod \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " Oct 31 20:11:44.628598 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.628547 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-kube-api-access-zgvq8\" (UniqueName: \"kubernetes.io/projected/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-kube-api-access-zgvq8\") pod \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\" (UID: \"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217\") " Oct 31 20:11:44.628654 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.628601 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-home" pod "0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217" (UID: "0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:11:44.628800 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.628749 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-model-cache" pod "0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217" (UID: "0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:11:44.629016 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.628826 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tmp-dir" pod "0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217" (UID: "0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:11:44.629095 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.629028 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:44.629095 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.629049 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:44.629095 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.629066 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:44.630676 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.630646 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-dshm" pod "0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217" (UID: "0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:11:44.630792 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.630765 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-kube-api-access-zgvq8" pod "0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217" (UID: "0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217"). InnerVolumeSpecName "kube-api-access-zgvq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:11:44.630915 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.630895 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tls-certs" pod "0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217" (UID: "0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:11:44.730098 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.730045 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgvq8\" (UniqueName: \"kubernetes.io/projected/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-kube-api-access-zgvq8\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:44.730098 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.730099 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:44.730098 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:44.730114 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:45.456031 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:45.455998 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" Oct 31 20:11:45.456496 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:45.455997 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4" event={"ID":"0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217","Type":"ContainerDied","Data":"d996683f23cff6ec8280ccb9d04bff8341c8bd8de77f899bd4f96c14058d55a4"} Oct 31 20:11:45.456496 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:45.456120 6107 scope.go:122] "RemoveContainer" containerID="af5eaa5a040246e813717af210ee13cd767a2a44e768a7d3896a7eb827965328" Oct 31 20:11:45.609999 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:45.609961 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4"] Oct 31 20:11:45.615414 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:45.615387 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a89a58/test-model-a89a58-kserve-78bcc66955-7dfr4"] Oct 31 20:11:46.487437 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:46.487403 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217" path="/var/lib/kubelet/pods/0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217/volumes" Oct 31 20:11:47.615983 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:47.615940 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc"] Oct 31 20:11:47.616477 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:47.616203 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" podUID="82ccf1df-3361-4057-980e-1b38a5fd5ea7" containerName="maas-api" containerID="cri-o://859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2" gracePeriod=30 Oct 31 20:11:47.883133 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:47.883069 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:11:47.965068 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:47.965032 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ccf1df-3361-4057-980e-1b38a5fd5ea7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ccf1df-3361-4057-980e-1b38a5fd5ea7-openshift-service-ca\") pod \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " Oct 31 20:11:47.965289 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:47.965216 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-metrics-certs\") pod \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " Oct 31 20:11:47.965289 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:47.965244 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-maas-api-tls\") pod \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " Oct 31 20:11:47.965289 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:47.965266 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ccf1df-3361-4057-980e-1b38a5fd5ea7-kube-api-access-qzw9k\" (UniqueName: \"kubernetes.io/projected/82ccf1df-3361-4057-980e-1b38a5fd5ea7-kube-api-access-qzw9k\") pod \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\" (UID: \"82ccf1df-3361-4057-980e-1b38a5fd5ea7\") " Oct 31 20:11:47.965512 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:47.965484 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ccf1df-3361-4057-980e-1b38a5fd5ea7-openshift-service-ca" pod "82ccf1df-3361-4057-980e-1b38a5fd5ea7" (UID: "82ccf1df-3361-4057-980e-1b38a5fd5ea7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:11:47.965759 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:47.965724 6107 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ccf1df-3361-4057-980e-1b38a5fd5ea7-openshift-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:47.967631 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:47.967594 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-maas-api-tls" pod "82ccf1df-3361-4057-980e-1b38a5fd5ea7" (UID: "82ccf1df-3361-4057-980e-1b38a5fd5ea7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:11:47.967745 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:47.967696 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-metrics-certs" pod "82ccf1df-3361-4057-980e-1b38a5fd5ea7" (UID: "82ccf1df-3361-4057-980e-1b38a5fd5ea7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:11:47.967844 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:47.967827 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ccf1df-3361-4057-980e-1b38a5fd5ea7-kube-api-access-qzw9k" pod "82ccf1df-3361-4057-980e-1b38a5fd5ea7" (UID: "82ccf1df-3361-4057-980e-1b38a5fd5ea7"). InnerVolumeSpecName "kube-api-access-qzw9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:11:48.066652 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:48.066616 6107 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-metrics-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:48.066652 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:48.066650 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ccf1df-3361-4057-980e-1b38a5fd5ea7-maas-api-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:48.066870 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:48.066664 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzw9k\" (UniqueName: \"kubernetes.io/projected/82ccf1df-3361-4057-980e-1b38a5fd5ea7-kube-api-access-qzw9k\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:48.475993 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:48.475954 6107 generic.go:356] "Generic (PLEG): container finished" podID="82ccf1df-3361-4057-980e-1b38a5fd5ea7" containerID="859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2" exitCode=0 Oct 31 20:11:48.476175 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:48.476008 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" Oct 31 20:11:48.476175 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:48.476040 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" event={"ID":"82ccf1df-3361-4057-980e-1b38a5fd5ea7","Type":"ContainerDied","Data":"859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2"} Oct 31 20:11:48.476175 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:48.476100 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc" event={"ID":"82ccf1df-3361-4057-980e-1b38a5fd5ea7","Type":"ContainerDied","Data":"bd52c59b2ca8522a64de8a31d7bd8280ea8338271a6f199828e68291691c487f"} Oct 31 20:11:48.476175 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:48.476121 6107 scope.go:122] "RemoveContainer" containerID="859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2" Oct 31 20:11:48.487127 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:48.487068 6107 scope.go:122] "RemoveContainer" containerID="859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2" Oct 31 20:11:48.487358 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:11:48.487339 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2\": container with ID starting with 859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2 not found: ID does not exist" containerID="859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2" Oct 31 20:11:48.487428 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:48.487366 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2"} err="failed to get container status \"859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2\": rpc error: code = NotFound desc = could not find container \"859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2\": container with ID starting with 859b1398e6bf3483f27256e36fafea3df1642692acc9d1a0c5ab52f78a0a14b2 not found: ID does not exist" Oct 31 20:11:48.635977 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:48.635937 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc"] Oct 31 20:11:48.681989 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:48.681954 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd95d7-85696b7d8f-cqswc"] Oct 31 20:11:50.405116 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:11:50.405070 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:11:50.495435 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:50.495395 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ccf1df-3361-4057-980e-1b38a5fd5ea7" path="/var/lib/kubelet/pods/82ccf1df-3361-4057-980e-1b38a5fd5ea7/volumes" Oct 31 20:11:56.688281 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:56.688242 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf"] Oct 31 20:11:56.688680 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:11:56.688516 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" podUID="5921ad77-6716-48a1-aac2-3c9a83497866" containerName="istio-proxy" containerID="cri-o://0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99" gracePeriod=30 Oct 31 20:12:01.948487 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:01.948460 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:12:02.129312 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.129278 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-data\") pod \"5921ad77-6716-48a1-aac2-3c9a83497866\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " Oct 31 20:12:02.129503 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.129377 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-certs\") pod \"5921ad77-6716-48a1-aac2-3c9a83497866\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " Oct 31 20:12:02.129503 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.129442 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5921ad77-6716-48a1-aac2-3c9a83497866-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5921ad77-6716-48a1-aac2-3c9a83497866-istio-podinfo\") pod \"5921ad77-6716-48a1-aac2-3c9a83497866\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " Oct 31 20:12:02.129503 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.129494 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-envoy\") pod \"5921ad77-6716-48a1-aac2-3c9a83497866\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " Oct 31 20:12:02.129660 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.129570 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-socket\") pod \"5921ad77-6716-48a1-aac2-3c9a83497866\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " Oct 31 20:12:02.129660 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.129613 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-kube-api-access-xlx4s\" (UniqueName: \"kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-kube-api-access-xlx4s\") pod \"5921ad77-6716-48a1-aac2-3c9a83497866\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " Oct 31 20:12:02.129731 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.129669 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5921ad77-6716-48a1-aac2-3c9a83497866-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5921ad77-6716-48a1-aac2-3c9a83497866-istiod-ca-cert\") pod \"5921ad77-6716-48a1-aac2-3c9a83497866\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " Oct 31 20:12:02.129731 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.129714 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-credential-socket\") pod \"5921ad77-6716-48a1-aac2-3c9a83497866\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " Oct 31 20:12:02.129802 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.129752 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-istio-token\" (UniqueName: \"kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-istio-token\") pod \"5921ad77-6716-48a1-aac2-3c9a83497866\" (UID: \"5921ad77-6716-48a1-aac2-3c9a83497866\") " Oct 31 20:12:02.130786 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.130424 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5921ad77-6716-48a1-aac2-3c9a83497866-istiod-ca-cert" pod "5921ad77-6716-48a1-aac2-3c9a83497866" (UID: "5921ad77-6716-48a1-aac2-3c9a83497866"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:12:02.130786 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.130443 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-credential-socket" pod "5921ad77-6716-48a1-aac2-3c9a83497866" (UID: "5921ad77-6716-48a1-aac2-3c9a83497866"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:02.130786 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.130626 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-certs" pod "5921ad77-6716-48a1-aac2-3c9a83497866" (UID: "5921ad77-6716-48a1-aac2-3c9a83497866"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:02.131057 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.130789 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-data" pod "5921ad77-6716-48a1-aac2-3c9a83497866" (UID: "5921ad77-6716-48a1-aac2-3c9a83497866"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:02.131213 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.131165 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-socket" pod "5921ad77-6716-48a1-aac2-3c9a83497866" (UID: "5921ad77-6716-48a1-aac2-3c9a83497866"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:02.132697 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.132671 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-istio-token" pod "5921ad77-6716-48a1-aac2-3c9a83497866" (UID: "5921ad77-6716-48a1-aac2-3c9a83497866"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:12:02.132960 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.132929 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-envoy" pod "5921ad77-6716-48a1-aac2-3c9a83497866" (UID: "5921ad77-6716-48a1-aac2-3c9a83497866"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:02.133073 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.132988 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-kube-api-access-xlx4s" pod "5921ad77-6716-48a1-aac2-3c9a83497866" (UID: "5921ad77-6716-48a1-aac2-3c9a83497866"). InnerVolumeSpecName "kube-api-access-xlx4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:12:02.133378 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.133356 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5921ad77-6716-48a1-aac2-3c9a83497866-istio-podinfo" pod "5921ad77-6716-48a1-aac2-3c9a83497866" (UID: "5921ad77-6716-48a1-aac2-3c9a83497866"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:12:02.231465 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.231431 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5921ad77-6716-48a1-aac2-3c9a83497866-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.231465 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.231464 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.231706 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.231476 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.231706 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.231490 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlx4s\" (UniqueName: \"kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-kube-api-access-xlx4s\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.231706 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.231503 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5921ad77-6716-48a1-aac2-3c9a83497866-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.231706 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.231514 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.231706 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.231525 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5921ad77-6716-48a1-aac2-3c9a83497866-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.231706 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.231537 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.231706 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.231549 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5921ad77-6716-48a1-aac2-3c9a83497866-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.570200 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.570154 6107 generic.go:356] "Generic (PLEG): container finished" podID="5921ad77-6716-48a1-aac2-3c9a83497866" containerID="0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99" exitCode=0 Oct 31 20:12:02.570396 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.570212 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" Oct 31 20:12:02.570396 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.570267 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" event={"ID":"5921ad77-6716-48a1-aac2-3c9a83497866","Type":"ContainerDied","Data":"0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99"} Oct 31 20:12:02.570396 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.570343 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf" event={"ID":"5921ad77-6716-48a1-aac2-3c9a83497866","Type":"ContainerDied","Data":"9d1b77a7f891c1601246e031cd989fd2986161d753ca48499cb8ce58cc20d1eb"} Oct 31 20:12:02.570396 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.570371 6107 scope.go:122] "RemoveContainer" containerID="0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99" Oct 31 20:12:02.580208 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.580189 6107 scope.go:122] "RemoveContainer" containerID="0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99" Oct 31 20:12:02.580496 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:12:02.580472 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99\": container with ID starting with 0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99 not found: ID does not exist" containerID="0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99" Oct 31 20:12:02.580569 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.580508 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99"} err="failed to get container status \"0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99\": rpc error: code = NotFound desc = could not find container \"0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99\": container with ID starting with 0d4c76925f488d6a38c676442565d5d4bd7e4f2be9a90c9d27a32089cbffaf99 not found: ID does not exist" Oct 31 20:12:02.768393 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.768299 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf"] Oct 31 20:12:02.772179 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:02.772150 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qsztf"] Oct 31 20:12:04.486753 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:04.486722 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5921ad77-6716-48a1-aac2-3c9a83497866" path="/var/lib/kubelet/pods/5921ad77-6716-48a1-aac2-3c9a83497866/volumes" Oct 31 20:12:20.408602 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:12:20.408559 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:12:39.515914 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:39.515857 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq"] Oct 31 20:12:39.616568 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:39.616530 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6"] Oct 31 20:12:44.517128 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.517070 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" podUID="bed8c1b4-a002-4087-8539-fd8a7c1ce7d3" containerName="payload-processing" containerID="cri-o://ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f" gracePeriod=25 Oct 31 20:12:44.616969 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.616915 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" podUID="784d6b85-dad0-4fb8-a9cd-153ec9f270d2" containerName="payload-pre-processing" containerID="cri-o://5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466" gracePeriod=25 Oct 31 20:12:44.778152 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.778098 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" Oct 31 20:12:44.812579 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.812539 6107 generic.go:356] "Generic (PLEG): container finished" podID="bed8c1b4-a002-4087-8539-fd8a7c1ce7d3" containerID="ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f" exitCode=0 Oct 31 20:12:44.812756 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.812585 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" Oct 31 20:12:44.812756 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.812614 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" event={"ID":"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3","Type":"ContainerDied","Data":"ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f"} Oct 31 20:12:44.812756 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.812671 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq" event={"ID":"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3","Type":"ContainerDied","Data":"d084220e4fe6c0372ade0113d94823f3991d80aee4934fa02851ce20f4f009ae"} Oct 31 20:12:44.812756 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.812699 6107 scope.go:122] "RemoveContainer" containerID="ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f" Oct 31 20:12:44.822205 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.822181 6107 scope.go:122] "RemoveContainer" containerID="ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f" Oct 31 20:12:44.822507 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:12:44.822489 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f\": container with ID starting with ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f not found: ID does not exist" containerID="ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f" Oct 31 20:12:44.822591 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.822512 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f"} err="failed to get container status \"ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f\": rpc error: code = NotFound desc = could not find container \"ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f\": container with ID starting with ab3498ed01dfb4e01aac79941d3e19d305b1fa923eaaeca57101d02f5e34d42f not found: ID does not exist" Oct 31 20:12:44.909638 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.909603 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-kube-api-access-qvgzj\" (UniqueName: \"kubernetes.io/projected/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-kube-api-access-qvgzj\") pod \"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3\" (UID: \"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3\") " Oct 31 20:12:44.909829 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.909649 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-plugins-config-volume\") pod \"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3\" (UID: \"bed8c1b4-a002-4087-8539-fd8a7c1ce7d3\") " Oct 31 20:12:44.910093 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.910072 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-plugins-config-volume" pod "bed8c1b4-a002-4087-8539-fd8a7c1ce7d3" (UID: "bed8c1b4-a002-4087-8539-fd8a7c1ce7d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:12:44.912079 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:44.912057 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-kube-api-access-qvgzj" pod "bed8c1b4-a002-4087-8539-fd8a7c1ce7d3" (UID: "bed8c1b4-a002-4087-8539-fd8a7c1ce7d3"). InnerVolumeSpecName "kube-api-access-qvgzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:12:45.010911 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:45.010864 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvgzj\" (UniqueName: \"kubernetes.io/projected/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-kube-api-access-qvgzj\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:45.011081 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:45.010921 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:45.223275 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:45.223240 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq"] Oct 31 20:12:45.227410 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:45.227384 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef44dd-559455ccf7-fjwrq"] Oct 31 20:12:45.416836 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:45.416761 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w"] Oct 31 20:12:45.426317 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:45.426287 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4"] Oct 31 20:12:45.436047 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:45.436017 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2"] Oct 31 20:12:46.486579 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:46.486543 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed8c1b4-a002-4087-8539-fd8a7c1ce7d3" path="/var/lib/kubelet/pods/bed8c1b4-a002-4087-8539-fd8a7c1ce7d3/volumes" Oct 31 20:12:48.036829 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:48.036752 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" podUID="784d6b85-dad0-4fb8-a9cd-153ec9f270d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:12:50.404862 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:12:50.404823 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:12:52.493795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:52.493762 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq"] Oct 31 20:12:57.495086 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:57.495039 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" podUID="b05e3c93-830d-40fa-b368-1b8f23413bd8" containerName="payload-processing" containerID="cri-o://2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753" gracePeriod=25 Oct 31 20:12:58.037239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:12:58.037201 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" podUID="784d6b85-dad0-4fb8-a9cd-153ec9f270d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:13:00.440619 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.440581 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" containerID="cri-o://dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a" gracePeriod=45 Oct 31 20:13:00.455356 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.455317 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" containerID="cri-o://07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e" gracePeriod=45 Oct 31 20:13:00.455477 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.455327 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" containerID="cri-o://6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f" gracePeriod=45 Oct 31 20:13:00.730463 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.730435 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 20:13:00.751428 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.751406 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 20:13:00.754667 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.754648 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 20:13:00.886048 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886017 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-model-cache\") pod \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " Oct 31 20:13:00.886048 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886060 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tls-certs\" (UniqueName: \"kubernetes.io/secret/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tls-certs\") pod \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " Oct 31 20:13:00.886311 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886084 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tmp-dir\") pod \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " Oct 31 20:13:00.886311 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886107 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-model-cache\") pod \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " Oct 31 20:13:00.886311 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886127 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-model-cache\") pod \"160c1547-6b7b-4d74-9041-b3eca6e20445\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " Oct 31 20:13:00.886311 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886175 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-tmp-dir\") pod \"160c1547-6b7b-4d74-9041-b3eca6e20445\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " Oct 31 20:13:00.886311 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886271 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-home\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-home\") pod \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " Oct 31 20:13:00.886311 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886308 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tls-certs\" (UniqueName: \"kubernetes.io/secret/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tls-certs\") pod \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " Oct 31 20:13:00.886629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886326 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-model-cache" pod "df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" (UID: "df9b0180-a73d-4dcb-a8d0-7709b3fc6f04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:00.886629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886348 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-dshm\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-dshm\") pod \"160c1547-6b7b-4d74-9041-b3eca6e20445\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " Oct 31 20:13:00.886629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886373 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-dshm\") pod \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " Oct 31 20:13:00.886629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886401 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-kube-api-access-jnvnz\" (UniqueName: \"kubernetes.io/projected/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-kube-api-access-jnvnz\") pod \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " Oct 31 20:13:00.886629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886434 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/160c1547-6b7b-4d74-9041-b3eca6e20445-tls-certs\" (UniqueName: \"kubernetes.io/secret/160c1547-6b7b-4d74-9041-b3eca6e20445-tls-certs\") pod \"160c1547-6b7b-4d74-9041-b3eca6e20445\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " Oct 31 20:13:00.886629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886448 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-model-cache" pod "1702ff79-4331-4a28-9ea8-63cc3d5d5a50" (UID: "1702ff79-4331-4a28-9ea8-63cc3d5d5a50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:00.886629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886491 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160c1547-6b7b-4d74-9041-b3eca6e20445-kube-api-access-vff7d\" (UniqueName: \"kubernetes.io/projected/160c1547-6b7b-4d74-9041-b3eca6e20445-kube-api-access-vff7d\") pod \"160c1547-6b7b-4d74-9041-b3eca6e20445\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " Oct 31 20:13:00.886629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886532 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-kube-api-access-xtmbf\" (UniqueName: \"kubernetes.io/projected/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-kube-api-access-xtmbf\") pod \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\" (UID: \"1702ff79-4331-4a28-9ea8-63cc3d5d5a50\") " Oct 31 20:13:00.886629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886549 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-home" pod "1702ff79-4331-4a28-9ea8-63cc3d5d5a50" (UID: "1702ff79-4331-4a28-9ea8-63cc3d5d5a50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:00.886629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886559 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-home\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-home\") pod \"160c1547-6b7b-4d74-9041-b3eca6e20445\" (UID: \"160c1547-6b7b-4d74-9041-b3eca6e20445\") " Oct 31 20:13:00.886629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886591 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-dshm\") pod \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " Oct 31 20:13:00.886629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886621 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-home\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-home\") pod \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " Oct 31 20:13:00.887281 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886695 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tmp-dir\") pod \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\" (UID: \"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04\") " Oct 31 20:13:00.887281 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886940 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tmp-dir" pod "1702ff79-4331-4a28-9ea8-63cc3d5d5a50" (UID: "1702ff79-4331-4a28-9ea8-63cc3d5d5a50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:00.887281 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.887059 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-tmp-dir" pod "160c1547-6b7b-4d74-9041-b3eca6e20445" (UID: "160c1547-6b7b-4d74-9041-b3eca6e20445"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:00.887281 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.887217 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.887281 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.887239 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.887281 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.887254 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.887281 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.887267 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.887281 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.887281 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.887694 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.886945 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-model-cache" pod "160c1547-6b7b-4d74-9041-b3eca6e20445" (UID: "160c1547-6b7b-4d74-9041-b3eca6e20445"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:00.888823 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.888130 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-home" pod "df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" (UID: "df9b0180-a73d-4dcb-a8d0-7709b3fc6f04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:00.888823 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.888341 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tmp-dir" pod "df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" (UID: "df9b0180-a73d-4dcb-a8d0-7709b3fc6f04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:00.888823 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.888504 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-home" pod "160c1547-6b7b-4d74-9041-b3eca6e20445" (UID: "160c1547-6b7b-4d74-9041-b3eca6e20445"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:00.890101 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.890080 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tls-certs" pod "df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" (UID: "df9b0180-a73d-4dcb-a8d0-7709b3fc6f04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:13:00.891275 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.891254 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-dshm" pod "160c1547-6b7b-4d74-9041-b3eca6e20445" (UID: "160c1547-6b7b-4d74-9041-b3eca6e20445"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:00.891386 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.891298 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160c1547-6b7b-4d74-9041-b3eca6e20445-kube-api-access-vff7d" pod "160c1547-6b7b-4d74-9041-b3eca6e20445" (UID: "160c1547-6b7b-4d74-9041-b3eca6e20445"). InnerVolumeSpecName "kube-api-access-vff7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:00.891386 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.891321 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-dshm" pod "1702ff79-4331-4a28-9ea8-63cc3d5d5a50" (UID: "1702ff79-4331-4a28-9ea8-63cc3d5d5a50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:00.891386 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.891336 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tls-certs" pod "1702ff79-4331-4a28-9ea8-63cc3d5d5a50" (UID: "1702ff79-4331-4a28-9ea8-63cc3d5d5a50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:13:00.891569 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.891460 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160c1547-6b7b-4d74-9041-b3eca6e20445-tls-certs" pod "160c1547-6b7b-4d74-9041-b3eca6e20445" (UID: "160c1547-6b7b-4d74-9041-b3eca6e20445"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:13:00.892475 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.892457 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-kube-api-access-xtmbf" pod "1702ff79-4331-4a28-9ea8-63cc3d5d5a50" (UID: "1702ff79-4331-4a28-9ea8-63cc3d5d5a50"). InnerVolumeSpecName "kube-api-access-xtmbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:00.892764 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.892740 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-kube-api-access-jnvnz" pod "df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" (UID: "df9b0180-a73d-4dcb-a8d0-7709b3fc6f04"). InnerVolumeSpecName "kube-api-access-jnvnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:00.893049 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.893029 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-dshm" pod "df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" (UID: "df9b0180-a73d-4dcb-a8d0-7709b3fc6f04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:00.906764 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.906739 6107 generic.go:356] "Generic (PLEG): container finished" podID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerID="dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a" exitCode=0 Oct 31 20:13:00.906875 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.906790 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" Oct 31 20:13:00.906875 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.906814 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" event={"ID":"160c1547-6b7b-4d74-9041-b3eca6e20445","Type":"ContainerDied","Data":"dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a"} Oct 31 20:13:00.906875 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.906851 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w" event={"ID":"160c1547-6b7b-4d74-9041-b3eca6e20445","Type":"ContainerDied","Data":"c54ef5ce04bf769bbf3abac6437fa1f060365dcb31b4a2baaab63c6e3b96b2d0"} Oct 31 20:13:00.907044 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.906867 6107 scope.go:122] "RemoveContainer" containerID="dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a" Oct 31 20:13:00.908350 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.908326 6107 generic.go:356] "Generic (PLEG): container finished" podID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerID="6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f" exitCode=0 Oct 31 20:13:00.908447 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.908371 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" Oct 31 20:13:00.908447 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.908386 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" event={"ID":"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04","Type":"ContainerDied","Data":"6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f"} Oct 31 20:13:00.908447 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.908420 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2" event={"ID":"df9b0180-a73d-4dcb-a8d0-7709b3fc6f04","Type":"ContainerDied","Data":"e330f019c03fffb9f19eeb09be6ba39fce8a4e17eac3e3e6e3001212309de4b1"} Oct 31 20:13:00.909810 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.909780 6107 generic.go:356] "Generic (PLEG): container finished" podID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerID="07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e" exitCode=0 Oct 31 20:13:00.909933 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.909809 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" event={"ID":"1702ff79-4331-4a28-9ea8-63cc3d5d5a50","Type":"ContainerDied","Data":"07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e"} Oct 31 20:13:00.909933 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.909824 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" Oct 31 20:13:00.909933 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.909834 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4" event={"ID":"1702ff79-4331-4a28-9ea8-63cc3d5d5a50","Type":"ContainerDied","Data":"5bcf5aa65bd939c0e073f7994747b3212a486d9fbd7a6e71d721a9469bf9ecbc"} Oct 31 20:13:00.917093 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.917076 6107 scope.go:122] "RemoveContainer" containerID="dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a" Oct 31 20:13:00.917732 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:13:00.917707 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a\": container with ID starting with dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a not found: ID does not exist" containerID="dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a" Oct 31 20:13:00.917809 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.917732 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a"} err="failed to get container status \"dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a\": rpc error: code = NotFound desc = could not find container \"dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a\": container with ID starting with dd61b4ec601a9f03ab47ed5c22e45f4ca3ff980e5d8d295f25b58b48e496093a not found: ID does not exist" Oct 31 20:13:00.917809 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.917754 6107 scope.go:122] "RemoveContainer" containerID="6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f" Oct 31 20:13:00.926449 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.926432 6107 scope.go:122] "RemoveContainer" containerID="6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f" Oct 31 20:13:00.926693 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:13:00.926676 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f\": container with ID starting with 6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f not found: ID does not exist" containerID="6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f" Oct 31 20:13:00.926751 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.926699 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f"} err="failed to get container status \"6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f\": rpc error: code = NotFound desc = could not find container \"6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f\": container with ID starting with 6c2b5511b388fac1128c4f6c099c745edbd620a0385b52e7e5b000761075be2f not found: ID does not exist" Oct 31 20:13:00.926751 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.926714 6107 scope.go:122] "RemoveContainer" containerID="07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e" Oct 31 20:13:00.934496 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.934480 6107 scope.go:122] "RemoveContainer" containerID="07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e" Oct 31 20:13:00.934773 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:13:00.934752 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e\": container with ID starting with 07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e not found: ID does not exist" containerID="07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e" Oct 31 20:13:00.934825 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.934779 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e"} err="failed to get container status \"07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e\": rpc error: code = NotFound desc = could not find container \"07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e\": container with ID starting with 07c05047ea2c4a6b9e3b25c6bd3f6ab503c693187e8124e3a60522b71355487e not found: ID does not exist" Oct 31 20:13:00.987919 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.987863 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.987919 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.987917 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.987919 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.987926 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.988229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.987935 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.988229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.987943 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.988229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.987954 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.988229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.987963 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.988229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.987971 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.988229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.987980 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnvnz\" (UniqueName: \"kubernetes.io/projected/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04-kube-api-access-jnvnz\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.988229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.987989 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160c1547-6b7b-4d74-9041-b3eca6e20445-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.988229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.987997 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vff7d\" (UniqueName: \"kubernetes.io/projected/160c1547-6b7b-4d74-9041-b3eca6e20445-kube-api-access-vff7d\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.988229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.988008 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtmbf\" (UniqueName: \"kubernetes.io/projected/1702ff79-4331-4a28-9ea8-63cc3d5d5a50-kube-api-access-xtmbf\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:00.988229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:00.988017 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160c1547-6b7b-4d74-9041-b3eca6e20445-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:01.166176 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:01.166143 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2"] Oct 31 20:13:01.169072 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:01.169044 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/simulator-w2-967dee-kserve-588c564d67-bssg2"] Oct 31 20:13:01.282610 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:01.282526 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4"] Oct 31 20:13:01.287582 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:01.287556 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/premium-w2-967dee-kserve-54f5c97dc8-kbhb4"] Oct 31 20:13:01.389091 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:01.389059 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w"] Oct 31 20:13:01.393553 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:01.393522 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-967dee/distinct-w2-967dee-kserve-c55c6f8f-brv8w"] Oct 31 20:13:01.543000 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:01.542915 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" podUID="b05e3c93-830d-40fa-b368-1b8f23413bd8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:13:02.485768 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:02.485734 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" path="/var/lib/kubelet/pods/160c1547-6b7b-4d74-9041-b3eca6e20445/volumes" Oct 31 20:13:02.486186 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:02.486172 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" path="/var/lib/kubelet/pods/1702ff79-4331-4a28-9ea8-63cc3d5d5a50/volumes" Oct 31 20:13:02.486560 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:02.486548 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" path="/var/lib/kubelet/pods/df9b0180-a73d-4dcb-a8d0-7709b3fc6f04/volumes" Oct 31 20:13:04.893517 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:04.893490 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" Oct 31 20:13:04.939998 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:04.939961 6107 generic.go:356] "Generic (PLEG): container finished" podID="784d6b85-dad0-4fb8-a9cd-153ec9f270d2" containerID="5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466" exitCode=0 Oct 31 20:13:04.940156 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:04.940006 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" Oct 31 20:13:04.940156 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:04.940044 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" event={"ID":"784d6b85-dad0-4fb8-a9cd-153ec9f270d2","Type":"ContainerDied","Data":"5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466"} Oct 31 20:13:04.940156 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:04.940096 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6" event={"ID":"784d6b85-dad0-4fb8-a9cd-153ec9f270d2","Type":"ContainerDied","Data":"c575e9162e8dad58271ca3297e921972c04677ebcf233f986dfe69eeaf4cb2ff"} Oct 31 20:13:04.940156 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:04.940119 6107 scope.go:122] "RemoveContainer" containerID="5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466" Oct 31 20:13:04.949447 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:04.949429 6107 scope.go:122] "RemoveContainer" containerID="5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466" Oct 31 20:13:04.949741 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:13:04.949723 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466\": container with ID starting with 5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466 not found: ID does not exist" containerID="5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466" Oct 31 20:13:04.949799 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:04.949750 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466"} err="failed to get container status \"5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466\": rpc error: code = NotFound desc = could not find container \"5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466\": container with ID starting with 5972974ee24a936003bc64a67489697ff67008aa671c1e9e99af5842acd5c466 not found: ID does not exist" Oct 31 20:13:05.031373 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:05.031342 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-plugins-config-volume\") pod \"784d6b85-dad0-4fb8-a9cd-153ec9f270d2\" (UID: \"784d6b85-dad0-4fb8-a9cd-153ec9f270d2\") " Oct 31 20:13:05.031532 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:05.031403 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-kube-api-access-wn6r7\" (UniqueName: \"kubernetes.io/projected/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-kube-api-access-wn6r7\") pod \"784d6b85-dad0-4fb8-a9cd-153ec9f270d2\" (UID: \"784d6b85-dad0-4fb8-a9cd-153ec9f270d2\") " Oct 31 20:13:05.031762 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:05.031735 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-plugins-config-volume" pod "784d6b85-dad0-4fb8-a9cd-153ec9f270d2" (UID: "784d6b85-dad0-4fb8-a9cd-153ec9f270d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:05.033457 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:05.033435 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-kube-api-access-wn6r7" pod "784d6b85-dad0-4fb8-a9cd-153ec9f270d2" (UID: "784d6b85-dad0-4fb8-a9cd-153ec9f270d2"). InnerVolumeSpecName "kube-api-access-wn6r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:05.132478 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:05.132398 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn6r7\" (UniqueName: \"kubernetes.io/projected/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-kube-api-access-wn6r7\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:05.132478 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:05.132427 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784d6b85-dad0-4fb8-a9cd-153ec9f270d2-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:05.351466 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:05.351428 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6"] Oct 31 20:13:05.354442 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:05.354410 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef44dd-6f56664cdc-qp4z6"] Oct 31 20:13:06.486579 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:06.486533 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784d6b85-dad0-4fb8-a9cd-153ec9f270d2" path="/var/lib/kubelet/pods/784d6b85-dad0-4fb8-a9cd-153ec9f270d2/volumes" Oct 31 20:13:11.543471 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:11.543424 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" podUID="b05e3c93-830d-40fa-b368-1b8f23413bd8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:13:17.744813 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:17.744787 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" Oct 31 20:13:17.863341 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:17.863264 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b05e3c93-830d-40fa-b368-1b8f23413bd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05e3c93-830d-40fa-b368-1b8f23413bd8-plugins-config-volume\") pod \"b05e3c93-830d-40fa-b368-1b8f23413bd8\" (UID: \"b05e3c93-830d-40fa-b368-1b8f23413bd8\") " Oct 31 20:13:17.863341 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:17.863324 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05e3c93-830d-40fa-b368-1b8f23413bd8-kube-api-access-9swfh\" (UniqueName: \"kubernetes.io/projected/b05e3c93-830d-40fa-b368-1b8f23413bd8-kube-api-access-9swfh\") pod \"b05e3c93-830d-40fa-b368-1b8f23413bd8\" (UID: \"b05e3c93-830d-40fa-b368-1b8f23413bd8\") " Oct 31 20:13:17.863652 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:17.863625 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05e3c93-830d-40fa-b368-1b8f23413bd8-plugins-config-volume" pod "b05e3c93-830d-40fa-b368-1b8f23413bd8" (UID: "b05e3c93-830d-40fa-b368-1b8f23413bd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:17.865371 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:17.865335 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05e3c93-830d-40fa-b368-1b8f23413bd8-kube-api-access-9swfh" pod "b05e3c93-830d-40fa-b368-1b8f23413bd8" (UID: "b05e3c93-830d-40fa-b368-1b8f23413bd8"). InnerVolumeSpecName "kube-api-access-9swfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:17.964543 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:17.964506 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05e3c93-830d-40fa-b368-1b8f23413bd8-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:17.964543 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:17.964543 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9swfh\" (UniqueName: \"kubernetes.io/projected/b05e3c93-830d-40fa-b368-1b8f23413bd8-kube-api-access-9swfh\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:18.016150 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:18.016119 6107 generic.go:356] "Generic (PLEG): container finished" podID="b05e3c93-830d-40fa-b368-1b8f23413bd8" containerID="2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753" exitCode=0 Oct 31 20:13:18.016312 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:18.016162 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" Oct 31 20:13:18.016312 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:18.016173 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" event={"ID":"b05e3c93-830d-40fa-b368-1b8f23413bd8","Type":"ContainerDied","Data":"2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753"} Oct 31 20:13:18.016312 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:18.016214 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq" event={"ID":"b05e3c93-830d-40fa-b368-1b8f23413bd8","Type":"ContainerDied","Data":"a5895634d93844bc97562128b13cabf3511dd77ff8c2251a4fc7625cbf9d147b"} Oct 31 20:13:18.016312 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:18.016241 6107 scope.go:122] "RemoveContainer" containerID="2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753" Oct 31 20:13:18.026036 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:18.026018 6107 scope.go:122] "RemoveContainer" containerID="2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753" Oct 31 20:13:18.026293 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:13:18.026273 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753\": container with ID starting with 2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753 not found: ID does not exist" containerID="2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753" Oct 31 20:13:18.026343 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:18.026303 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753"} err="failed to get container status \"2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753\": rpc error: code = NotFound desc = could not find container \"2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753\": container with ID starting with 2146cfc01796261358fc6a23251717e3cd6f39f4b3d34fc68a198d3b02c7d753 not found: ID does not exist" Oct 31 20:13:18.169380 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:18.169261 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq"] Oct 31 20:13:18.172666 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:18.172630 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-967dee-659784ff55-8zhqq"] Oct 31 20:13:18.486848 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:18.486817 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05e3c93-830d-40fa-b368-1b8f23413bd8" path="/var/lib/kubelet/pods/b05e3c93-830d-40fa-b368-1b8f23413bd8/volumes" Oct 31 20:13:19.274407 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:19.274372 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt"] Oct 31 20:13:19.377426 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:19.377396 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n"] Oct 31 20:13:20.406848 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:13:20.406808 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:13:24.275537 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.275480 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" podUID="b0829bac-192a-4311-b773-2896d3b96490" containerName="payload-processing" containerID="cri-o://aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a" gracePeriod=25 Oct 31 20:13:24.377969 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.377928 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" podUID="702ddccb-4d46-487e-ba61-1e40f396893f" containerName="payload-pre-processing" containerID="cri-o://a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74" gracePeriod=25 Oct 31 20:13:24.522608 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.522582 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" Oct 31 20:13:24.631311 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.631285 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" Oct 31 20:13:24.637379 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.637351 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0829bac-192a-4311-b773-2896d3b96490-kube-api-access-ft7sh\" (UniqueName: \"kubernetes.io/projected/b0829bac-192a-4311-b773-2896d3b96490-kube-api-access-ft7sh\") pod \"b0829bac-192a-4311-b773-2896d3b96490\" (UID: \"b0829bac-192a-4311-b773-2896d3b96490\") " Oct 31 20:13:24.637508 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.637389 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0829bac-192a-4311-b773-2896d3b96490-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0829bac-192a-4311-b773-2896d3b96490-plugins-config-volume\") pod \"b0829bac-192a-4311-b773-2896d3b96490\" (UID: \"b0829bac-192a-4311-b773-2896d3b96490\") " Oct 31 20:13:24.637769 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.637742 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0829bac-192a-4311-b773-2896d3b96490-plugins-config-volume" pod "b0829bac-192a-4311-b773-2896d3b96490" (UID: "b0829bac-192a-4311-b773-2896d3b96490"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:24.639767 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.639742 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0829bac-192a-4311-b773-2896d3b96490-kube-api-access-ft7sh" pod "b0829bac-192a-4311-b773-2896d3b96490" (UID: "b0829bac-192a-4311-b773-2896d3b96490"). InnerVolumeSpecName "kube-api-access-ft7sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:24.738229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.738202 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702ddccb-4d46-487e-ba61-1e40f396893f-kube-api-access-zb4ld\" (UniqueName: \"kubernetes.io/projected/702ddccb-4d46-487e-ba61-1e40f396893f-kube-api-access-zb4ld\") pod \"702ddccb-4d46-487e-ba61-1e40f396893f\" (UID: \"702ddccb-4d46-487e-ba61-1e40f396893f\") " Oct 31 20:13:24.738394 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.738335 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702ddccb-4d46-487e-ba61-1e40f396893f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702ddccb-4d46-487e-ba61-1e40f396893f-plugins-config-volume\") pod \"702ddccb-4d46-487e-ba61-1e40f396893f\" (UID: \"702ddccb-4d46-487e-ba61-1e40f396893f\") " Oct 31 20:13:24.738596 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.738584 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft7sh\" (UniqueName: \"kubernetes.io/projected/b0829bac-192a-4311-b773-2896d3b96490-kube-api-access-ft7sh\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:24.738633 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.738599 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0829bac-192a-4311-b773-2896d3b96490-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:24.738703 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.738681 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702ddccb-4d46-487e-ba61-1e40f396893f-plugins-config-volume" pod "702ddccb-4d46-487e-ba61-1e40f396893f" (UID: "702ddccb-4d46-487e-ba61-1e40f396893f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:24.740471 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.740448 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702ddccb-4d46-487e-ba61-1e40f396893f-kube-api-access-zb4ld" pod "702ddccb-4d46-487e-ba61-1e40f396893f" (UID: "702ddccb-4d46-487e-ba61-1e40f396893f"). InnerVolumeSpecName "kube-api-access-zb4ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:24.839943 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.839848 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb4ld\" (UniqueName: \"kubernetes.io/projected/702ddccb-4d46-487e-ba61-1e40f396893f-kube-api-access-zb4ld\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:24.839943 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:24.839874 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702ddccb-4d46-487e-ba61-1e40f396893f-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:25.061384 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.061345 6107 generic.go:356] "Generic (PLEG): container finished" podID="b0829bac-192a-4311-b773-2896d3b96490" containerID="aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a" exitCode=0 Oct 31 20:13:25.061584 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.061394 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" Oct 31 20:13:25.061584 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.061431 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" event={"ID":"b0829bac-192a-4311-b773-2896d3b96490","Type":"ContainerDied","Data":"aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a"} Oct 31 20:13:25.061584 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.061486 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt" event={"ID":"b0829bac-192a-4311-b773-2896d3b96490","Type":"ContainerDied","Data":"1f4dabb88844384e1dd3aa23af99dfb7108fad7f6a6aa3066abc8bfe7eb97b1c"} Oct 31 20:13:25.061584 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.061511 6107 scope.go:122] "RemoveContainer" containerID="aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a" Oct 31 20:13:25.062818 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.062791 6107 generic.go:356] "Generic (PLEG): container finished" podID="702ddccb-4d46-487e-ba61-1e40f396893f" containerID="a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74" exitCode=0 Oct 31 20:13:25.062946 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.062839 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" Oct 31 20:13:25.062946 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.062910 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" event={"ID":"702ddccb-4d46-487e-ba61-1e40f396893f","Type":"ContainerDied","Data":"a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74"} Oct 31 20:13:25.063078 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.062951 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n" event={"ID":"702ddccb-4d46-487e-ba61-1e40f396893f","Type":"ContainerDied","Data":"383cd46cbdd44459bee80052ef7103bdf834895212c06673cf4d42f359233cea"} Oct 31 20:13:25.071853 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.071832 6107 scope.go:122] "RemoveContainer" containerID="aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a" Oct 31 20:13:25.072156 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:13:25.072139 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a\": container with ID starting with aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a not found: ID does not exist" containerID="aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a" Oct 31 20:13:25.072248 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.072169 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a"} err="failed to get container status \"aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a\": rpc error: code = NotFound desc = could not find container \"aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a\": container with ID starting with aed1ef40deee9528ca9e094e909c0b85674ab7dfc58836fcd8bd0e0e6f74830a not found: ID does not exist" Oct 31 20:13:25.072248 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.072190 6107 scope.go:122] "RemoveContainer" containerID="a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74" Oct 31 20:13:25.080096 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.080079 6107 scope.go:122] "RemoveContainer" containerID="a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74" Oct 31 20:13:25.080340 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:13:25.080323 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74\": container with ID starting with a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74 not found: ID does not exist" containerID="a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74" Oct 31 20:13:25.080390 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.080347 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74"} err="failed to get container status \"a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74\": rpc error: code = NotFound desc = could not find container \"a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74\": container with ID starting with a92114817ea5be707cc7ebd2da47d64aa4b3e11ccebbb8b41f6f450866cf2e74 not found: ID does not exist" Oct 31 20:13:25.215173 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.215083 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n"] Oct 31 20:13:25.223584 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.223550 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-872fee-545d96457f-gxb2n"] Oct 31 20:13:25.325199 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.325169 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt"] Oct 31 20:13:25.327376 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:25.327350 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-872fee-8d959c987-jjflt"] Oct 31 20:13:26.487386 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:26.487345 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702ddccb-4d46-487e-ba61-1e40f396893f" path="/var/lib/kubelet/pods/702ddccb-4d46-487e-ba61-1e40f396893f/volumes" Oct 31 20:13:26.488012 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:26.487906 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0829bac-192a-4311-b773-2896d3b96490" path="/var/lib/kubelet/pods/b0829bac-192a-4311-b773-2896d3b96490/volumes" Oct 31 20:13:40.992265 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:40.992234 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d"] Oct 31 20:13:40.993271 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:40.992483 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" containerID="cri-o://d9469f74a1f815d598e3136f38ac346b69fff7455454e769fe2029ddc75f91e0" gracePeriod=30 Oct 31 20:13:43.239840 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:43.239802 6107 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:13:43.240257 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:43.239865 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:13:46.187193 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.187162 6107 generic.go:356] "Generic (PLEG): container finished" podID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerID="d9469f74a1f815d598e3136f38ac346b69fff7455454e769fe2029ddc75f91e0" exitCode=0 Oct 31 20:13:46.187606 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.187219 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" event={"ID":"dd0ed64f-59a6-4a0a-9293-0d435b303fb5","Type":"ContainerDied","Data":"d9469f74a1f815d598e3136f38ac346b69fff7455454e769fe2029ddc75f91e0"} Oct 31 20:13:46.255463 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.255433 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 20:13:46.372433 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.372355 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-certs\") pod \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " Oct 31 20:13:46.372433 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.372405 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istiod-ca-cert\") pod \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " Oct 31 20:13:46.372653 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.372446 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-envoy\") pod \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " Oct 31 20:13:46.372653 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.372466 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-socket\") pod \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " Oct 31 20:13:46.372653 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.372490 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-credential-socket\") pod \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " Oct 31 20:13:46.372653 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.372532 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-kube-api-access-jfb5z\" (UniqueName: \"kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-kube-api-access-jfb5z\") pod \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " Oct 31 20:13:46.372653 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.372561 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-token\" (UniqueName: \"kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-token\") pod \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " Oct 31 20:13:46.372653 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.372601 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-podinfo\") pod \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " Oct 31 20:13:46.373154 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.372709 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-data\") pod \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\" (UID: \"dd0ed64f-59a6-4a0a-9293-0d435b303fb5\") " Oct 31 20:13:46.373154 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.372748 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-certs" pod "dd0ed64f-59a6-4a0a-9293-0d435b303fb5" (UID: "dd0ed64f-59a6-4a0a-9293-0d435b303fb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:46.373154 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.372765 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-socket" pod "dd0ed64f-59a6-4a0a-9293-0d435b303fb5" (UID: "dd0ed64f-59a6-4a0a-9293-0d435b303fb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:46.373154 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.372805 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-credential-socket" pod "dd0ed64f-59a6-4a0a-9293-0d435b303fb5" (UID: "dd0ed64f-59a6-4a0a-9293-0d435b303fb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:46.373154 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.373109 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:46.373154 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.373128 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:46.373154 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.373144 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:46.373154 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.373125 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-data" pod "dd0ed64f-59a6-4a0a-9293-0d435b303fb5" (UID: "dd0ed64f-59a6-4a0a-9293-0d435b303fb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:46.373604 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.373137 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istiod-ca-cert" pod "dd0ed64f-59a6-4a0a-9293-0d435b303fb5" (UID: "dd0ed64f-59a6-4a0a-9293-0d435b303fb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:46.374978 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.374950 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-kube-api-access-jfb5z" pod "dd0ed64f-59a6-4a0a-9293-0d435b303fb5" (UID: "dd0ed64f-59a6-4a0a-9293-0d435b303fb5"). InnerVolumeSpecName "kube-api-access-jfb5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:46.374978 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.374969 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-envoy" pod "dd0ed64f-59a6-4a0a-9293-0d435b303fb5" (UID: "dd0ed64f-59a6-4a0a-9293-0d435b303fb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:46.375136 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.375113 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-podinfo" pod "dd0ed64f-59a6-4a0a-9293-0d435b303fb5" (UID: "dd0ed64f-59a6-4a0a-9293-0d435b303fb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:13:46.375136 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.375115 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-token" pod "dd0ed64f-59a6-4a0a-9293-0d435b303fb5" (UID: "dd0ed64f-59a6-4a0a-9293-0d435b303fb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:46.474369 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.474334 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:46.474369 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.474365 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:46.474369 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.474375 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:46.474617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.474384 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfb5z\" (UniqueName: \"kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-kube-api-access-jfb5z\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:46.474617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.474393 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:46.474617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:46.474401 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0ed64f-59a6-4a0a-9293-0d435b303fb5-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:47.193871 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:47.193839 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" Oct 31 20:13:47.194344 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:47.193865 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d" event={"ID":"dd0ed64f-59a6-4a0a-9293-0d435b303fb5","Type":"ContainerDied","Data":"3eea62097e526d1479a81ca3c341a17a74c41e67b781c986c0978c6f6412713c"} Oct 31 20:13:47.194344 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:47.193969 6107 scope.go:122] "RemoveContainer" containerID="d9469f74a1f815d598e3136f38ac346b69fff7455454e769fe2029ddc75f91e0" Oct 31 20:13:47.391312 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:47.391275 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d"] Oct 31 20:13:47.396700 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:47.396664 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-wld4d"] Oct 31 20:13:48.485417 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:13:48.485369 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" path="/var/lib/kubelet/pods/dd0ed64f-59a6-4a0a-9293-0d435b303fb5/volumes" Oct 31 20:13:50.402250 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:13:50.402218 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:14:10.184060 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:10.183976 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr"] Oct 31 20:14:10.184508 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:10.184176 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" podUID="46f0be40-d904-455b-9113-956835869a3d" containerName="istio-proxy" containerID="cri-o://7b49a81d025c4710481f551d04ce0fb93739200510b3250db0105ac183d7d549" gracePeriod=30 Oct 31 20:14:10.229942 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:10.229909 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg"] Oct 31 20:14:10.230133 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:10.230112 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerName="istio-proxy" containerID="cri-o://41eb49ab14ce295242f2bbcbd90f3ea279274f6dd03c1a4347dd248e219569f8" gracePeriod=30 Oct 31 20:14:12.411036 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:12.410990 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc"] Oct 31 20:14:12.455931 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:12.455873 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg"] Oct 31 20:14:15.363094 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.363062 6107 generic.go:356] "Generic (PLEG): container finished" podID="46f0be40-d904-455b-9113-956835869a3d" containerID="7b49a81d025c4710481f551d04ce0fb93739200510b3250db0105ac183d7d549" exitCode=0 Oct 31 20:14:15.363784 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.363757 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" event={"ID":"46f0be40-d904-455b-9113-956835869a3d","Type":"ContainerDied","Data":"7b49a81d025c4710481f551d04ce0fb93739200510b3250db0105ac183d7d549"} Oct 31 20:14:15.365949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.365922 6107 generic.go:356] "Generic (PLEG): container finished" podID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerID="41eb49ab14ce295242f2bbcbd90f3ea279274f6dd03c1a4347dd248e219569f8" exitCode=0 Oct 31 20:14:15.366070 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.365980 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" event={"ID":"b6fd63df-0cfe-4329-980d-f0383e0406ef","Type":"ContainerDied","Data":"41eb49ab14ce295242f2bbcbd90f3ea279274f6dd03c1a4347dd248e219569f8"} Oct 31 20:14:15.483136 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.483112 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:14:15.514073 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.514047 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:14:15.565641 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565615 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-socket\") pod \"46f0be40-d904-455b-9113-956835869a3d\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " Oct 31 20:14:15.565804 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565650 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-kube-api-access-djg9n\" (UniqueName: \"kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-kube-api-access-djg9n\") pod \"46f0be40-d904-455b-9113-956835869a3d\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " Oct 31 20:14:15.565804 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565689 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-credential-socket\") pod \"46f0be40-d904-455b-9113-956835869a3d\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " Oct 31 20:14:15.565804 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565708 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6fd63df-0cfe-4329-980d-f0383e0406ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fd63df-0cfe-4329-980d-f0383e0406ef-istiod-ca-cert\") pod \"b6fd63df-0cfe-4329-980d-f0383e0406ef\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " Oct 31 20:14:15.565804 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565731 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-istio-token\" (UniqueName: \"kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-istio-token\") pod \"46f0be40-d904-455b-9113-956835869a3d\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " Oct 31 20:14:15.565804 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565747 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-envoy\") pod \"b6fd63df-0cfe-4329-980d-f0383e0406ef\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " Oct 31 20:14:15.565804 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565772 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-certs\") pod \"46f0be40-d904-455b-9113-956835869a3d\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " Oct 31 20:14:15.565804 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565806 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-kube-api-access-lkdwr\" (UniqueName: \"kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-kube-api-access-lkdwr\") pod \"b6fd63df-0cfe-4329-980d-f0383e0406ef\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " Oct 31 20:14:15.566239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565834 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46f0be40-d904-455b-9113-956835869a3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f0be40-d904-455b-9113-956835869a3d-istio-podinfo\") pod \"46f0be40-d904-455b-9113-956835869a3d\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " Oct 31 20:14:15.566239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565909 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-token\" (UniqueName: \"kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-token\") pod \"b6fd63df-0cfe-4329-980d-f0383e0406ef\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " Oct 31 20:14:15.566239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565942 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-data\") pod \"46f0be40-d904-455b-9113-956835869a3d\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " Oct 31 20:14:15.566239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565953 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-socket" pod "46f0be40-d904-455b-9113-956835869a3d" (UID: "46f0be40-d904-455b-9113-956835869a3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:15.566239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.565978 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-data\") pod \"b6fd63df-0cfe-4329-980d-f0383e0406ef\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " Oct 31 20:14:15.566239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.566046 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-certs\") pod \"b6fd63df-0cfe-4329-980d-f0383e0406ef\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " Oct 31 20:14:15.566239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.566069 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-credential-socket" pod "46f0be40-d904-455b-9113-956835869a3d" (UID: "46f0be40-d904-455b-9113-956835869a3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:15.566239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.566121 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f0be40-d904-455b-9113-956835869a3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f0be40-d904-455b-9113-956835869a3d-istiod-ca-cert\") pod \"46f0be40-d904-455b-9113-956835869a3d\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " Oct 31 20:14:15.566239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.566148 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6fd63df-0cfe-4329-980d-f0383e0406ef-istiod-ca-cert" pod "b6fd63df-0cfe-4329-980d-f0383e0406ef" (UID: "b6fd63df-0cfe-4329-980d-f0383e0406ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:15.566778 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.566426 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-data" pod "b6fd63df-0cfe-4329-980d-f0383e0406ef" (UID: "b6fd63df-0cfe-4329-980d-f0383e0406ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:15.566778 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.566577 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-certs" pod "46f0be40-d904-455b-9113-956835869a3d" (UID: "46f0be40-d904-455b-9113-956835869a3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:15.566778 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.566681 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-certs" pod "b6fd63df-0cfe-4329-980d-f0383e0406ef" (UID: "b6fd63df-0cfe-4329-980d-f0383e0406ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:15.566778 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.566694 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-podinfo\") pod \"b6fd63df-0cfe-4329-980d-f0383e0406ef\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " Oct 31 20:14:15.566778 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.566717 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f0be40-d904-455b-9113-956835869a3d-istiod-ca-cert" pod "46f0be40-d904-455b-9113-956835869a3d" (UID: "46f0be40-d904-455b-9113-956835869a3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:15.566778 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.566737 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-socket\") pod \"b6fd63df-0cfe-4329-980d-f0383e0406ef\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " Oct 31 20:14:15.567081 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.566787 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-credential-socket\") pod \"b6fd63df-0cfe-4329-980d-f0383e0406ef\" (UID: \"b6fd63df-0cfe-4329-980d-f0383e0406ef\") " Oct 31 20:14:15.567081 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.566826 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-envoy\") pod \"46f0be40-d904-455b-9113-956835869a3d\" (UID: \"46f0be40-d904-455b-9113-956835869a3d\") " Oct 31 20:14:15.567081 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.567028 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-socket" pod "b6fd63df-0cfe-4329-980d-f0383e0406ef" (UID: "b6fd63df-0cfe-4329-980d-f0383e0406ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:15.567302 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.567282 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f0be40-d904-455b-9113-956835869a3d-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.567370 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.567306 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.567370 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.567321 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.567370 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.567336 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.567370 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.567351 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fd63df-0cfe-4329-980d-f0383e0406ef-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.567585 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.567379 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.567585 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.567402 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.567585 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.567417 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.567749 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.567609 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-credential-socket" pod "b6fd63df-0cfe-4329-980d-f0383e0406ef" (UID: "b6fd63df-0cfe-4329-980d-f0383e0406ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:15.567749 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.567680 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-data" pod "46f0be40-d904-455b-9113-956835869a3d" (UID: "46f0be40-d904-455b-9113-956835869a3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:15.568459 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.568431 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-envoy" pod "b6fd63df-0cfe-4329-980d-f0383e0406ef" (UID: "b6fd63df-0cfe-4329-980d-f0383e0406ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:15.568867 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.568844 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-kube-api-access-djg9n" pod "46f0be40-d904-455b-9113-956835869a3d" (UID: "46f0be40-d904-455b-9113-956835869a3d"). InnerVolumeSpecName "kube-api-access-djg9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:15.568979 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.568894 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-istio-token" pod "46f0be40-d904-455b-9113-956835869a3d" (UID: "46f0be40-d904-455b-9113-956835869a3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:15.569096 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.569080 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-kube-api-access-lkdwr" pod "b6fd63df-0cfe-4329-980d-f0383e0406ef" (UID: "b6fd63df-0cfe-4329-980d-f0383e0406ef"). InnerVolumeSpecName "kube-api-access-lkdwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:15.569185 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.569168 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-podinfo" pod "b6fd63df-0cfe-4329-980d-f0383e0406ef" (UID: "b6fd63df-0cfe-4329-980d-f0383e0406ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:14:15.569316 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.569301 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-token" pod "b6fd63df-0cfe-4329-980d-f0383e0406ef" (UID: "b6fd63df-0cfe-4329-980d-f0383e0406ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:15.569640 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.569622 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-envoy" pod "46f0be40-d904-455b-9113-956835869a3d" (UID: "46f0be40-d904-455b-9113-956835869a3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:15.569819 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.569803 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46f0be40-d904-455b-9113-956835869a3d-istio-podinfo" pod "46f0be40-d904-455b-9113-956835869a3d" (UID: "46f0be40-d904-455b-9113-956835869a3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:14:15.668687 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.668653 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djg9n\" (UniqueName: \"kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-kube-api-access-djg9n\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.668687 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.668680 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f0be40-d904-455b-9113-956835869a3d-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.668687 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.668690 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.668949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.668699 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkdwr\" (UniqueName: \"kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-kube-api-access-lkdwr\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.668949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.668708 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f0be40-d904-455b-9113-956835869a3d-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.668949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.668716 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.668949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.668725 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.668949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.668734 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fd63df-0cfe-4329-980d-f0383e0406ef-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.668949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.668742 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fd63df-0cfe-4329-980d-f0383e0406ef-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:15.668949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:15.668750 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f0be40-d904-455b-9113-956835869a3d-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:16.371798 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:16.371766 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" Oct 31 20:14:16.372302 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:16.371796 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr" event={"ID":"46f0be40-d904-455b-9113-956835869a3d","Type":"ContainerDied","Data":"e441e77723bbf124e0ca37813d6f2d3af8aa36069bcf28329c985c94c16541b6"} Oct 31 20:14:16.372302 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:16.371864 6107 scope.go:122] "RemoveContainer" containerID="7b49a81d025c4710481f551d04ce0fb93739200510b3250db0105ac183d7d549" Oct 31 20:14:16.373243 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:16.373209 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" event={"ID":"b6fd63df-0cfe-4329-980d-f0383e0406ef","Type":"ContainerDied","Data":"d5de269e56ff79e0982d1a90ed2648b0838dcf3c8f5b269f5e1ed2ab2837da83"} Oct 31 20:14:16.373322 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:16.373216 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg" Oct 31 20:14:16.382710 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:16.382689 6107 scope.go:122] "RemoveContainer" containerID="41eb49ab14ce295242f2bbcbd90f3ea279274f6dd03c1a4347dd248e219569f8" Oct 31 20:14:16.524013 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:16.523979 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr"] Oct 31 20:14:16.569549 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:16.569519 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xcrlr"] Oct 31 20:14:16.629916 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:16.629875 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg"] Oct 31 20:14:16.633713 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:16.633685 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gb7lg"] Oct 31 20:14:18.485386 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:18.485314 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46f0be40-d904-455b-9113-956835869a3d" path="/var/lib/kubelet/pods/46f0be40-d904-455b-9113-956835869a3d/volumes" Oct 31 20:14:18.485774 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:18.485747 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" path="/var/lib/kubelet/pods/b6fd63df-0cfe-4329-980d-f0383e0406ef/volumes" Oct 31 20:14:19.473374 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:19.473341 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267"] Oct 31 20:14:19.585108 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:19.585069 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr"] Oct 31 20:14:20.403775 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:14:20.403740 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:14:24.474621 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:24.474570 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" podUID="66dba4ff-acee-4801-9e65-dd0198d23537" containerName="payload-processing" containerID="cri-o://81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194" gracePeriod=25 Oct 31 20:14:24.586131 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:24.586074 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" podUID="de419c01-12e0-4606-a362-666c83e6c7ee" containerName="payload-pre-processing" containerID="cri-o://2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c" gracePeriod=25 Oct 31 20:14:27.431110 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.431047 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" containerID="cri-o://9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262" gracePeriod=45 Oct 31 20:14:27.474823 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.474773 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" containerID="cri-o://7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91" gracePeriod=45 Oct 31 20:14:27.718140 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.718116 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 20:14:27.738140 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.738120 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 20:14:27.785590 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.785564 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58f9cd11-9c06-442c-80fe-40637909f825-tls-certs\" (UniqueName: \"kubernetes.io/secret/58f9cd11-9c06-442c-80fe-40637909f825-tls-certs\") pod \"58f9cd11-9c06-442c-80fe-40637909f825\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " Oct 31 20:14:27.785698 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.785598 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-tmp-dir\") pod \"58f9cd11-9c06-442c-80fe-40637909f825\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " Oct 31 20:14:27.785698 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.785649 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-model-cache\") pod \"58f9cd11-9c06-442c-80fe-40637909f825\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " Oct 31 20:14:27.785771 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.785740 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f9cd11-9c06-442c-80fe-40637909f825-kube-api-access-zftfc\" (UniqueName: \"kubernetes.io/projected/58f9cd11-9c06-442c-80fe-40637909f825-kube-api-access-zftfc\") pod \"58f9cd11-9c06-442c-80fe-40637909f825\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " Oct 31 20:14:27.785818 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.785803 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-home\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-home\") pod \"58f9cd11-9c06-442c-80fe-40637909f825\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " Oct 31 20:14:27.785947 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.785852 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-dshm\") pod \"58f9cd11-9c06-442c-80fe-40637909f825\" (UID: \"58f9cd11-9c06-442c-80fe-40637909f825\") " Oct 31 20:14:27.786015 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.785940 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-tmp-dir" pod "58f9cd11-9c06-442c-80fe-40637909f825" (UID: "58f9cd11-9c06-442c-80fe-40637909f825"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:27.786015 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.785991 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-model-cache" pod "58f9cd11-9c06-442c-80fe-40637909f825" (UID: "58f9cd11-9c06-442c-80fe-40637909f825"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:27.786125 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.786038 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-home" pod "58f9cd11-9c06-442c-80fe-40637909f825" (UID: "58f9cd11-9c06-442c-80fe-40637909f825"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:27.786217 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.786197 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:27.786217 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.786215 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:27.786392 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.786232 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:27.787713 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.787690 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58f9cd11-9c06-442c-80fe-40637909f825-tls-certs" pod "58f9cd11-9c06-442c-80fe-40637909f825" (UID: "58f9cd11-9c06-442c-80fe-40637909f825"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:27.787803 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.787787 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-dshm" pod "58f9cd11-9c06-442c-80fe-40637909f825" (UID: "58f9cd11-9c06-442c-80fe-40637909f825"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:27.787845 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.787797 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f9cd11-9c06-442c-80fe-40637909f825-kube-api-access-zftfc" pod "58f9cd11-9c06-442c-80fe-40637909f825" (UID: "58f9cd11-9c06-442c-80fe-40637909f825"). InnerVolumeSpecName "kube-api-access-zftfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:27.886503 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.886470 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52f12f7d-97f3-463b-b84f-32129807de15-tls-certs\" (UniqueName: \"kubernetes.io/secret/52f12f7d-97f3-463b-b84f-32129807de15-tls-certs\") pod \"52f12f7d-97f3-463b-b84f-32129807de15\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " Oct 31 20:14:27.886679 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.886532 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-tmp-dir\") pod \"52f12f7d-97f3-463b-b84f-32129807de15\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " Oct 31 20:14:27.886679 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.886565 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f12f7d-97f3-463b-b84f-32129807de15-kube-api-access-qgfk4\" (UniqueName: \"kubernetes.io/projected/52f12f7d-97f3-463b-b84f-32129807de15-kube-api-access-qgfk4\") pod \"52f12f7d-97f3-463b-b84f-32129807de15\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " Oct 31 20:14:27.886679 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.886605 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-home\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-home\") pod \"52f12f7d-97f3-463b-b84f-32129807de15\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " Oct 31 20:14:27.886679 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.886673 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-dshm\") pod \"52f12f7d-97f3-463b-b84f-32129807de15\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " Oct 31 20:14:27.886874 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.886723 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-model-cache\") pod \"52f12f7d-97f3-463b-b84f-32129807de15\" (UID: \"52f12f7d-97f3-463b-b84f-32129807de15\") " Oct 31 20:14:27.887320 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.887070 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zftfc\" (UniqueName: \"kubernetes.io/projected/58f9cd11-9c06-442c-80fe-40637909f825-kube-api-access-zftfc\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:27.887320 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.887094 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f9cd11-9c06-442c-80fe-40637909f825-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:27.887320 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.887108 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58f9cd11-9c06-442c-80fe-40637909f825-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:27.887320 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.887246 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-home" pod "52f12f7d-97f3-463b-b84f-32129807de15" (UID: "52f12f7d-97f3-463b-b84f-32129807de15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:27.887320 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.887232 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-tmp-dir" pod "52f12f7d-97f3-463b-b84f-32129807de15" (UID: "52f12f7d-97f3-463b-b84f-32129807de15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:27.887320 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.887280 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-model-cache" pod "52f12f7d-97f3-463b-b84f-32129807de15" (UID: "52f12f7d-97f3-463b-b84f-32129807de15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:27.889520 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.889491 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f12f7d-97f3-463b-b84f-32129807de15-tls-certs" pod "52f12f7d-97f3-463b-b84f-32129807de15" (UID: "52f12f7d-97f3-463b-b84f-32129807de15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:27.889520 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.889505 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f12f7d-97f3-463b-b84f-32129807de15-kube-api-access-qgfk4" pod "52f12f7d-97f3-463b-b84f-32129807de15" (UID: "52f12f7d-97f3-463b-b84f-32129807de15"). InnerVolumeSpecName "kube-api-access-qgfk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:27.889719 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.889586 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-dshm" pod "52f12f7d-97f3-463b-b84f-32129807de15" (UID: "52f12f7d-97f3-463b-b84f-32129807de15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:27.988133 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.988102 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f12f7d-97f3-463b-b84f-32129807de15-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:27.988133 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.988130 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:27.988321 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.988142 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgfk4\" (UniqueName: \"kubernetes.io/projected/52f12f7d-97f3-463b-b84f-32129807de15-kube-api-access-qgfk4\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:27.988321 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.988152 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:27.988321 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.988160 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:27.988321 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:27.988168 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f12f7d-97f3-463b-b84f-32129807de15-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:28.441531 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.441455 6107 generic.go:356] "Generic (PLEG): container finished" podID="58f9cd11-9c06-442c-80fe-40637909f825" containerID="9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262" exitCode=0 Oct 31 20:14:28.441531 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.441513 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" Oct 31 20:14:28.442082 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.441536 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" event={"ID":"58f9cd11-9c06-442c-80fe-40637909f825","Type":"ContainerDied","Data":"9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262"} Oct 31 20:14:28.442082 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.441592 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc" event={"ID":"58f9cd11-9c06-442c-80fe-40637909f825","Type":"ContainerDied","Data":"a15bfc4cf8228e1edf9b015738421c681d7303ae1907408bf8dc3654cf7c838b"} Oct 31 20:14:28.442082 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.441616 6107 scope.go:122] "RemoveContainer" containerID="9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262" Oct 31 20:14:28.443042 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.443013 6107 generic.go:356] "Generic (PLEG): container finished" podID="52f12f7d-97f3-463b-b84f-32129807de15" containerID="7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91" exitCode=0 Oct 31 20:14:28.443106 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.443053 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" Oct 31 20:14:28.443166 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.443059 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" event={"ID":"52f12f7d-97f3-463b-b84f-32129807de15","Type":"ContainerDied","Data":"7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91"} Oct 31 20:14:28.443252 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.443193 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg" event={"ID":"52f12f7d-97f3-463b-b84f-32129807de15","Type":"ContainerDied","Data":"dd1a899f02f67617060e32d7faeffd9e7577612d430ff9a529e0bc993e18359f"} Oct 31 20:14:28.463994 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.463974 6107 scope.go:122] "RemoveContainer" containerID="9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262" Oct 31 20:14:28.464639 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:14:28.464616 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262\": container with ID starting with 9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262 not found: ID does not exist" containerID="9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262" Oct 31 20:14:28.464813 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.464787 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262"} err="failed to get container status \"9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262\": rpc error: code = NotFound desc = could not find container \"9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262\": container with ID starting with 9c65cc225459916270a4bf5fe65969c1969c1e764e03c2464bf375b00fa77262 not found: ID does not exist" Oct 31 20:14:28.464944 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.464931 6107 scope.go:122] "RemoveContainer" containerID="7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91" Oct 31 20:14:28.474675 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.474652 6107 scope.go:122] "RemoveContainer" containerID="7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91" Oct 31 20:14:28.474939 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:14:28.474920 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91\": container with ID starting with 7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91 not found: ID does not exist" containerID="7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91" Oct 31 20:14:28.475005 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.474951 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91"} err="failed to get container status \"7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91\": rpc error: code = NotFound desc = could not find container \"7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91\": container with ID starting with 7f63d308eb9e6480d88749128e3cf9abb36bb9d09497dcac9983ba34c24d3d91 not found: ID does not exist" Oct 31 20:14:28.496423 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.496391 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" podUID="de419c01-12e0-4606-a362-666c83e6c7ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:14:28.496576 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.496507 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" podUID="66dba4ff-acee-4801-9e65-dd0198d23537" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:14:28.604296 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.604263 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc"] Oct 31 20:14:28.607079 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.607050 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/distinct-2-w0-093b97-kserve-ddfbc56d8-v8smc"] Oct 31 20:14:28.708319 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.708234 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg"] Oct 31 20:14:28.710792 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:28.710763 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-093b97/premium-w0-093b97-kserve-5c89fcfd48-czvtg"] Oct 31 20:14:30.485761 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:30.485728 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f12f7d-97f3-463b-b84f-32129807de15" path="/var/lib/kubelet/pods/52f12f7d-97f3-463b-b84f-32129807de15/volumes" Oct 31 20:14:30.486177 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:30.486162 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f9cd11-9c06-442c-80fe-40637909f825" path="/var/lib/kubelet/pods/58f9cd11-9c06-442c-80fe-40637909f825/volumes" Oct 31 20:14:34.799547 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:34.799516 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd"] Oct 31 20:14:34.799982 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:34.799741 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" podUID="4831dbcb-ea5f-4862-8250-cbe24c6fbd81" containerName="maas-api" containerID="cri-o://f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3" gracePeriod=30 Oct 31 20:14:35.060316 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.060259 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 20:14:35.156616 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.156587 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-openshift-service-ca\") pod \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " Oct 31 20:14:35.156795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.156668 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-kube-api-access-q4zzv\" (UniqueName: \"kubernetes.io/projected/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-kube-api-access-q4zzv\") pod \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " Oct 31 20:14:35.156795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.156733 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-metrics-certs\") pod \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " Oct 31 20:14:35.156795 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.156771 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-maas-api-tls\") pod \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\" (UID: \"4831dbcb-ea5f-4862-8250-cbe24c6fbd81\") " Oct 31 20:14:35.157057 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.157034 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-openshift-service-ca" pod "4831dbcb-ea5f-4862-8250-cbe24c6fbd81" (UID: "4831dbcb-ea5f-4862-8250-cbe24c6fbd81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:35.158794 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.158770 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-maas-api-tls" pod "4831dbcb-ea5f-4862-8250-cbe24c6fbd81" (UID: "4831dbcb-ea5f-4862-8250-cbe24c6fbd81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:35.158945 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.158832 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-metrics-certs" pod "4831dbcb-ea5f-4862-8250-cbe24c6fbd81" (UID: "4831dbcb-ea5f-4862-8250-cbe24c6fbd81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:35.158945 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.158858 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-kube-api-access-q4zzv" pod "4831dbcb-ea5f-4862-8250-cbe24c6fbd81" (UID: "4831dbcb-ea5f-4862-8250-cbe24c6fbd81"). InnerVolumeSpecName "kube-api-access-q4zzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:35.258399 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.258365 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4zzv\" (UniqueName: \"kubernetes.io/projected/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-kube-api-access-q4zzv\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:35.258399 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.258392 6107 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-metrics-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:35.258399 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.258402 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-maas-api-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:35.258626 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.258411 6107 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4831dbcb-ea5f-4862-8250-cbe24c6fbd81-openshift-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:35.482726 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.482694 6107 generic.go:356] "Generic (PLEG): container finished" podID="4831dbcb-ea5f-4862-8250-cbe24c6fbd81" containerID="f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3" exitCode=0 Oct 31 20:14:35.482914 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.482739 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" Oct 31 20:14:35.482914 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.482770 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" event={"ID":"4831dbcb-ea5f-4862-8250-cbe24c6fbd81","Type":"ContainerDied","Data":"f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3"} Oct 31 20:14:35.482914 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.482822 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd" event={"ID":"4831dbcb-ea5f-4862-8250-cbe24c6fbd81","Type":"ContainerDied","Data":"fecfbbff1690bff51bd9a3b5c04e95f2c7a547d78c3f1a1a8e9ef9da01fc2388"} Oct 31 20:14:35.482914 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.482848 6107 scope.go:122] "RemoveContainer" containerID="f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3" Oct 31 20:14:35.491954 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.491935 6107 scope.go:122] "RemoveContainer" containerID="f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3" Oct 31 20:14:35.492196 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:14:35.492178 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3\": container with ID starting with f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3 not found: ID does not exist" containerID="f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3" Oct 31 20:14:35.492266 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.492201 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3"} err="failed to get container status \"f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3\": rpc error: code = NotFound desc = could not find container \"f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3\": container with ID starting with f1be00fa7e11cd93fe3a0ad74d309cde8c46ac2311e43ebd2964cf7b853cf1d3 not found: ID does not exist" Oct 31 20:14:35.662327 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.662292 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd"] Oct 31 20:14:35.667005 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:35.666980 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-093b97-859cb79975-kq8wd"] Oct 31 20:14:36.485426 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:36.485394 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4831dbcb-ea5f-4862-8250-cbe24c6fbd81" path="/var/lib/kubelet/pods/4831dbcb-ea5f-4862-8250-cbe24c6fbd81/volumes" Oct 31 20:14:38.496379 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:38.496343 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" podUID="66dba4ff-acee-4801-9e65-dd0198d23537" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:14:38.496787 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:38.496348 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" podUID="de419c01-12e0-4606-a362-666c83e6c7ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:14:44.727611 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.727581 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" Oct 31 20:14:44.746769 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.746699 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66dba4ff-acee-4801-9e65-dd0198d23537-kube-api-access-n7x6m\" (UniqueName: \"kubernetes.io/projected/66dba4ff-acee-4801-9e65-dd0198d23537-kube-api-access-n7x6m\") pod \"66dba4ff-acee-4801-9e65-dd0198d23537\" (UID: \"66dba4ff-acee-4801-9e65-dd0198d23537\") " Oct 31 20:14:44.746769 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.746767 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66dba4ff-acee-4801-9e65-dd0198d23537-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66dba4ff-acee-4801-9e65-dd0198d23537-plugins-config-volume\") pod \"66dba4ff-acee-4801-9e65-dd0198d23537\" (UID: \"66dba4ff-acee-4801-9e65-dd0198d23537\") " Oct 31 20:14:44.747227 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.747197 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66dba4ff-acee-4801-9e65-dd0198d23537-plugins-config-volume" pod "66dba4ff-acee-4801-9e65-dd0198d23537" (UID: "66dba4ff-acee-4801-9e65-dd0198d23537"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:44.748857 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.748835 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66dba4ff-acee-4801-9e65-dd0198d23537-kube-api-access-n7x6m" pod "66dba4ff-acee-4801-9e65-dd0198d23537" (UID: "66dba4ff-acee-4801-9e65-dd0198d23537"). InnerVolumeSpecName "kube-api-access-n7x6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:44.820492 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.820468 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" Oct 31 20:14:44.847198 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.847175 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de419c01-12e0-4606-a362-666c83e6c7ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de419c01-12e0-4606-a362-666c83e6c7ee-plugins-config-volume\") pod \"de419c01-12e0-4606-a362-666c83e6c7ee\" (UID: \"de419c01-12e0-4606-a362-666c83e6c7ee\") " Oct 31 20:14:44.847334 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.847253 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de419c01-12e0-4606-a362-666c83e6c7ee-kube-api-access-qk42v\" (UniqueName: \"kubernetes.io/projected/de419c01-12e0-4606-a362-666c83e6c7ee-kube-api-access-qk42v\") pod \"de419c01-12e0-4606-a362-666c83e6c7ee\" (UID: \"de419c01-12e0-4606-a362-666c83e6c7ee\") " Oct 31 20:14:44.847482 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.847470 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7x6m\" (UniqueName: \"kubernetes.io/projected/66dba4ff-acee-4801-9e65-dd0198d23537-kube-api-access-n7x6m\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:44.847522 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.847485 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66dba4ff-acee-4801-9e65-dd0198d23537-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:44.847621 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.847591 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de419c01-12e0-4606-a362-666c83e6c7ee-plugins-config-volume" pod "de419c01-12e0-4606-a362-666c83e6c7ee" (UID: "de419c01-12e0-4606-a362-666c83e6c7ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:44.849232 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.849212 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de419c01-12e0-4606-a362-666c83e6c7ee-kube-api-access-qk42v" pod "de419c01-12e0-4606-a362-666c83e6c7ee" (UID: "de419c01-12e0-4606-a362-666c83e6c7ee"). InnerVolumeSpecName "kube-api-access-qk42v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:44.948471 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.948438 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de419c01-12e0-4606-a362-666c83e6c7ee-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:44.948471 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:44.948472 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk42v\" (UniqueName: \"kubernetes.io/projected/de419c01-12e0-4606-a362-666c83e6c7ee-kube-api-access-qk42v\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:45.536376 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.536344 6107 generic.go:356] "Generic (PLEG): container finished" podID="de419c01-12e0-4606-a362-666c83e6c7ee" containerID="2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c" exitCode=0 Oct 31 20:14:45.536552 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.536389 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" Oct 31 20:14:45.536552 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.536432 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" event={"ID":"de419c01-12e0-4606-a362-666c83e6c7ee","Type":"ContainerDied","Data":"2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c"} Oct 31 20:14:45.536552 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.536495 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr" event={"ID":"de419c01-12e0-4606-a362-666c83e6c7ee","Type":"ContainerDied","Data":"5e623b328e078a5c2f640969bcf1c3dad8a12d7cb3a7fcf1510e4bbea6945706"} Oct 31 20:14:45.536552 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.536522 6107 scope.go:122] "RemoveContainer" containerID="2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c" Oct 31 20:14:45.537858 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.537834 6107 generic.go:356] "Generic (PLEG): container finished" podID="66dba4ff-acee-4801-9e65-dd0198d23537" containerID="81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194" exitCode=0 Oct 31 20:14:45.537948 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.537898 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" Oct 31 20:14:45.537948 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.537906 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" event={"ID":"66dba4ff-acee-4801-9e65-dd0198d23537","Type":"ContainerDied","Data":"81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194"} Oct 31 20:14:45.537948 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.537945 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267" event={"ID":"66dba4ff-acee-4801-9e65-dd0198d23537","Type":"ContainerDied","Data":"1331dd00ca0b4f06713512535a49048006ed1e52c09cdeed216150030d611bbe"} Oct 31 20:14:45.547090 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.546994 6107 scope.go:122] "RemoveContainer" containerID="2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c" Oct 31 20:14:45.547307 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:14:45.547281 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c\": container with ID starting with 2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c not found: ID does not exist" containerID="2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c" Oct 31 20:14:45.547376 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.547316 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c"} err="failed to get container status \"2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c\": rpc error: code = NotFound desc = could not find container \"2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c\": container with ID starting with 2b35af552ddeb68f50d7962db9abf4449d99c66736b5549852d0956b7d5b0c8c not found: ID does not exist" Oct 31 20:14:45.547376 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.547339 6107 scope.go:122] "RemoveContainer" containerID="81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194" Oct 31 20:14:45.555176 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.555161 6107 scope.go:122] "RemoveContainer" containerID="81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194" Oct 31 20:14:45.555419 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:14:45.555399 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194\": container with ID starting with 81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194 not found: ID does not exist" containerID="81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194" Oct 31 20:14:45.555478 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.555428 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194"} err="failed to get container status \"81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194\": rpc error: code = NotFound desc = could not find container \"81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194\": container with ID starting with 81283101ee4dd04d6ff7e379358d5cb037ce4429b13ee8ad5bd35e66ff367194 not found: ID does not exist" Oct 31 20:14:45.687286 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.687253 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr"] Oct 31 20:14:45.733314 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.733283 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-093b97-7848957df5-jf9fr"] Oct 31 20:14:45.833969 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.833892 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267"] Oct 31 20:14:45.837667 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:45.837640 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-093b97-65b87b7855-6d267"] Oct 31 20:14:46.486766 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:46.486720 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66dba4ff-acee-4801-9e65-dd0198d23537" path="/var/lib/kubelet/pods/66dba4ff-acee-4801-9e65-dd0198d23537/volumes" Oct 31 20:14:46.487221 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:14:46.487198 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de419c01-12e0-4606-a362-666c83e6c7ee" path="/var/lib/kubelet/pods/de419c01-12e0-4606-a362-666c83e6c7ee/volumes" Oct 31 20:14:50.402332 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:14:50.402293 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:15:10.636168 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636136 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8"] Oct 31 20:15:10.636545 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636494 6107 cpu_manager.go:414] "containerMap: removing container" podUID="702ddccb-4d46-487e-ba61-1e40f396893f" containerName="payload-pre-processing" Oct 31 20:15:10.636545 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636506 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702ddccb-4d46-487e-ba61-1e40f396893f" containerName="payload-pre-processing" Oct 31 20:15:10.636545 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636519 6107 cpu_manager.go:414] "containerMap: removing container" podUID="b0829bac-192a-4311-b773-2896d3b96490" containerName="payload-processing" Oct 31 20:15:10.636545 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636525 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0829bac-192a-4311-b773-2896d3b96490" containerName="payload-processing" Oct 31 20:15:10.636545 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636537 6107 cpu_manager.go:414] "containerMap: removing container" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerName="istio-proxy" Oct 31 20:15:10.636545 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636542 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerName="istio-proxy" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636553 6107 cpu_manager.go:414] "containerMap: removing container" podUID="5921ad77-6716-48a1-aac2-3c9a83497866" containerName="istio-proxy" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636558 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5921ad77-6716-48a1-aac2-3c9a83497866" containerName="istio-proxy" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636566 6107 cpu_manager.go:414] "containerMap: removing container" podUID="46f0be40-d904-455b-9113-956835869a3d" containerName="istio-proxy" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636572 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f0be40-d904-455b-9113-956835869a3d" containerName="istio-proxy" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636578 6107 cpu_manager.go:414] "containerMap: removing container" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636584 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636596 6107 cpu_manager.go:414] "containerMap: removing container" podUID="0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217" containerName="main" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636601 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217" containerName="main" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636607 6107 cpu_manager.go:414] "containerMap: removing container" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636611 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636617 6107 cpu_manager.go:414] "containerMap: removing container" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636621 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636628 6107 cpu_manager.go:414] "containerMap: removing container" podUID="4831dbcb-ea5f-4862-8250-cbe24c6fbd81" containerName="maas-api" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636633 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4831dbcb-ea5f-4862-8250-cbe24c6fbd81" containerName="maas-api" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636638 6107 cpu_manager.go:414] "containerMap: removing container" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636644 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636653 6107 cpu_manager.go:414] "containerMap: removing container" podUID="b05e3c93-830d-40fa-b368-1b8f23413bd8" containerName="payload-processing" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636658 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05e3c93-830d-40fa-b368-1b8f23413bd8" containerName="payload-processing" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636673 6107 cpu_manager.go:414] "containerMap: removing container" podUID="de419c01-12e0-4606-a362-666c83e6c7ee" containerName="payload-pre-processing" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636678 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de419c01-12e0-4606-a362-666c83e6c7ee" containerName="payload-pre-processing" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636688 6107 cpu_manager.go:414] "containerMap: removing container" podUID="784d6b85-dad0-4fb8-a9cd-153ec9f270d2" containerName="payload-pre-processing" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636693 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784d6b85-dad0-4fb8-a9cd-153ec9f270d2" containerName="payload-pre-processing" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636699 6107 cpu_manager.go:414] "containerMap: removing container" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636704 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636709 6107 cpu_manager.go:414] "containerMap: removing container" podUID="bed8c1b4-a002-4087-8539-fd8a7c1ce7d3" containerName="payload-processing" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636714 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed8c1b4-a002-4087-8539-fd8a7c1ce7d3" containerName="payload-processing" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636721 6107 cpu_manager.go:414] "containerMap: removing container" podUID="66dba4ff-acee-4801-9e65-dd0198d23537" containerName="payload-processing" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636725 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66dba4ff-acee-4801-9e65-dd0198d23537" containerName="payload-processing" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636732 6107 cpu_manager.go:414] "containerMap: removing container" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" Oct 31 20:15:10.636737 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636740 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636752 6107 cpu_manager.go:414] "containerMap: removing container" podUID="ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" containerName="istio-proxy" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636757 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" containerName="istio-proxy" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636765 6107 cpu_manager.go:414] "containerMap: removing container" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636770 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636777 6107 cpu_manager.go:414] "containerMap: removing container" podUID="82ccf1df-3361-4057-980e-1b38a5fd5ea7" containerName="maas-api" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636780 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ccf1df-3361-4057-980e-1b38a5fd5ea7" containerName="maas-api" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636786 6107 cpu_manager.go:414] "containerMap: removing container" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636790 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636843 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="4831dbcb-ea5f-4862-8250-cbe24c6fbd81" containerName="maas-api" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636851 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f638482-f128-45a1-8a74-3d9751414c81" containerName="main" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636858 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05e3c93-830d-40fa-b368-1b8f23413bd8" containerName="payload-processing" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636864 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="de419c01-12e0-4606-a362-666c83e6c7ee" containerName="payload-pre-processing" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636872 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="784d6b85-dad0-4fb8-a9cd-153ec9f270d2" containerName="payload-pre-processing" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636892 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f9cd11-9c06-442c-80fe-40637909f825" containerName="main" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636901 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed8c1b4-a002-4087-8539-fd8a7c1ce7d3" containerName="payload-processing" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636907 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="66dba4ff-acee-4801-9e65-dd0198d23537" containerName="payload-processing" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636912 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="160c1547-6b7b-4d74-9041-b3eca6e20445" containerName="main" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636917 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7d07ec-a5e8-4a93-b7d4-9a5764e73a79" containerName="istio-proxy" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636924 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636929 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ccf1df-3361-4057-980e-1b38a5fd5ea7" containerName="maas-api" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636935 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="1702ff79-4331-4a28-9ea8-63cc3d5d5a50" containerName="main" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636939 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0ed64f-59a6-4a0a-9293-0d435b303fb5" containerName="istio-proxy" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636944 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="702ddccb-4d46-487e-ba61-1e40f396893f" containerName="payload-pre-processing" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636949 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0829bac-192a-4311-b773-2896d3b96490" containerName="payload-processing" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636955 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6fd63df-0cfe-4329-980d-f0383e0406ef" containerName="istio-proxy" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636961 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="5921ad77-6716-48a1-aac2-3c9a83497866" containerName="istio-proxy" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636969 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f0be40-d904-455b-9113-956835869a3d" containerName="istio-proxy" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636977 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f12f7d-97f3-463b-b84f-32129807de15" containerName="main" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636984 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c8bf88c-41d0-4e8b-9f0c-5c9e6966f217" containerName="main" Oct 31 20:15:10.637617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.636989 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9b0180-a73d-4dcb-a8d0-7709b3fc6f04" containerName="main" Oct 31 20:15:10.640032 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.640016 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.736416 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.736382 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8"] Oct 31 20:15:10.800122 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.800088 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwpt\" (UniqueName: \"kubernetes.io/projected/a5360256-7648-404e-ae6c-e5556cca1ae3-kube-api-access-qgwpt\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.800300 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.800131 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5360256-7648-404e-ae6c-e5556cca1ae3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.800300 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.800242 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5360256-7648-404e-ae6c-e5556cca1ae3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.800300 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.800275 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.800402 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.800365 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.800402 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.800394 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.800472 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.800434 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.800503 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.800467 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5360256-7648-404e-ae6c-e5556cca1ae3-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.800537 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.800517 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901288 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901214 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901288 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901262 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901288 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901288 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901564 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901306 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901564 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901324 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5360256-7648-404e-ae6c-e5556cca1ae3-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901564 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901360 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901564 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901387 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwpt\" (UniqueName: \"kubernetes.io/projected/a5360256-7648-404e-ae6c-e5556cca1ae3-kube-api-access-qgwpt\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901564 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901419 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5360256-7648-404e-ae6c-e5556cca1ae3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901564 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901486 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5360256-7648-404e-ae6c-e5556cca1ae3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901868 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901629 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901868 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901684 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901868 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901791 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.901868 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.901859 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.902228 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.902211 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5360256-7648-404e-ae6c-e5556cca1ae3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.903639 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.903617 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5360256-7648-404e-ae6c-e5556cca1ae3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.903861 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.903842 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5360256-7648-404e-ae6c-e5556cca1ae3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.955275 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.955239 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5360256-7648-404e-ae6c-e5556cca1ae3-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:10.955421 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:10.955325 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwpt\" (UniqueName: \"kubernetes.io/projected/a5360256-7648-404e-ae6c-e5556cca1ae3-kube-api-access-qgwpt\") pod \"maas-default-gateway-openshift-default-58d898bd55-zjsq8\" (UID: \"a5360256-7648-404e-ae6c-e5556cca1ae3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:11.252735 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:11.252697 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:11.660439 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:11.660408 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8"] Oct 31 20:15:11.663124 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:15:11.663095 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5360256_7648_404e_ae6c_e5556cca1ae3.slice/crio-081d9686462b661eb16a3f7fee3c16ca23e0a0c311587b012b73a04d07e31212 WatchSource:0}: Error finding container 081d9686462b661eb16a3f7fee3c16ca23e0a0c311587b012b73a04d07e31212: Status 404 returned error can't find the container with id 081d9686462b661eb16a3f7fee3c16ca23e0a0c311587b012b73a04d07e31212 Oct 31 20:15:11.665176 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:11.665147 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:15:11.665234 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:11.665207 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:15:11.665279 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:11.665237 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:15:11.671458 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:11.671427 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" event={"ID":"a5360256-7648-404e-ae6c-e5556cca1ae3","Type":"ContainerStarted","Data":"081d9686462b661eb16a3f7fee3c16ca23e0a0c311587b012b73a04d07e31212"} Oct 31 20:15:12.678771 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:12.678728 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" event={"ID":"a5360256-7648-404e-ae6c-e5556cca1ae3","Type":"ContainerStarted","Data":"bc9376846e34c5803b75c0877ddd9661910aef78361cfb95ef20b7cfa459553b"} Oct 31 20:15:12.882784 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:12.882725 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" podStartSLOduration=2.882705058 podStartE2EDuration="2.882705058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:15:12.835329773 +0000 UTC m=+3484.468723682" watchObservedRunningTime="2026-10-31 20:15:12.882705058 +0000 UTC m=+3484.516098966" Oct 31 20:15:13.253769 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:13.253741 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:13.254870 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:13.254844 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zjsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 20:15:13.255008 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:13.254902 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" podUID="a5360256-7648-404e-ae6c-e5556cca1ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 20:15:14.253457 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:14.253423 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zjsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 20:15:14.253840 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:14.253502 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" podUID="a5360256-7648-404e-ae6c-e5556cca1ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 20:15:15.253690 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:15.253653 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zjsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 20:15:15.254330 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:15.253715 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" podUID="a5360256-7648-404e-ae6c-e5556cca1ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 20:15:16.253863 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:16.253829 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zjsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 20:15:16.254422 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:16.253967 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" podUID="a5360256-7648-404e-ae6c-e5556cca1ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 20:15:17.254058 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:17.253717 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zjsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 20:15:17.254058 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:17.253766 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" podUID="a5360256-7648-404e-ae6c-e5556cca1ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 20:15:18.253857 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:18.253817 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zjsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 20:15:18.254047 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:18.253873 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" podUID="a5360256-7648-404e-ae6c-e5556cca1ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 20:15:20.254392 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:20.254345 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zjsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:15:20.254824 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:20.254418 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" podUID="a5360256-7648-404e-ae6c-e5556cca1ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:15:20.402715 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:15:20.402679 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:15:21.296082 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:21.296043 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zjsq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:15:21.296527 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:21.296105 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" podUID="a5360256-7648-404e-ae6c-e5556cca1ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:15:21.630474 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:21.630393 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:21.630694 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:21.630671 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:21.631354 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:15:21.631336 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" Oct 31 20:15:50.403074 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:15:50.402992 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:16:20.403955 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:16:20.403911 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:16:41.308166 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.308129 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6"] Oct 31 20:16:41.308694 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.308518 6107 cpu_manager.go:414] "containerMap: removing container" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" Oct 31 20:16:41.308694 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.308530 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" Oct 31 20:16:41.308694 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.308590 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="704ab7da-672d-4ed0-8efa-7177dde795b0" containerName="istio-proxy" Oct 31 20:16:41.311863 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.311841 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.405507 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.405461 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6"] Oct 31 20:16:41.406079 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.406054 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.406244 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.406103 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.406244 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.406219 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5c9f\" (UniqueName: \"kubernetes.io/projected/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-kube-api-access-f5c9f\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.406392 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.406265 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.406392 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.406296 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.406392 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.406330 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.406392 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.406361 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.406392 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.406386 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.406637 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.406433 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.507281 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.507246 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.507281 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.507282 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.507606 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.507303 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.507606 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.507493 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.507606 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.507519 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.507606 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.507546 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.507821 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.507622 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.507821 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.507663 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.507821 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.507711 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5c9f\" (UniqueName: \"kubernetes.io/projected/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-kube-api-access-f5c9f\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.508051 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.507973 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.508051 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.507989 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.508169 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.508045 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.508244 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.508225 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.508386 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.508364 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.509944 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.509925 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.510081 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.510064 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.560221 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.560152 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.560364 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.560319 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5c9f\" (UniqueName: \"kubernetes.io/projected/7fb2c48a-65ce-436f-8e39-29d5f7aa0969-kube-api-access-f5c9f\") pod \"maas-default-gateway-openshift-default-58d898bd55-r67h6\" (UID: \"7fb2c48a-65ce-436f-8e39-29d5f7aa0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:41.624333 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:41.624299 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:42.035945 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:42.035913 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6"] Oct 31 20:16:42.044014 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:42.043976 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:16:42.044148 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:42.044045 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:16:42.044148 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:42.044088 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:16:42.146262 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:42.145992 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" event={"ID":"7fb2c48a-65ce-436f-8e39-29d5f7aa0969","Type":"ContainerStarted","Data":"bcd859964956dbb48c0ecb418326c93bf99f77b7ebe0d5c50d8f7aae6cef7f3c"} Oct 31 20:16:42.146262 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:42.146064 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" event={"ID":"7fb2c48a-65ce-436f-8e39-29d5f7aa0969","Type":"ContainerStarted","Data":"cc8642d8fbd832ee58ceeb8c5cb897a105f06be47ad6b48e5df990b217ff7229"} Oct 31 20:16:42.256118 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:42.256065 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" podStartSLOduration=1.256045912 podStartE2EDuration="1.256045912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:16:42.25467221 +0000 UTC m=+3573.888066128" watchObservedRunningTime="2026-10-31 20:16:42.256045912 +0000 UTC m=+3573.889439820" Oct 31 20:16:42.624413 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:42.624373 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:43.627776 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:43.627735 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r67h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:16:43.628385 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:43.628352 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" podUID="7fb2c48a-65ce-436f-8e39-29d5f7aa0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 20:16:44.625390 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:44.625349 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r67h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:16:44.625616 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:44.625419 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" podUID="7fb2c48a-65ce-436f-8e39-29d5f7aa0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 20:16:45.625328 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:45.625293 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r67h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:16:45.625803 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:45.625367 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" podUID="7fb2c48a-65ce-436f-8e39-29d5f7aa0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 20:16:46.628359 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:46.628309 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r67h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:16:46.628823 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:46.628396 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" podUID="7fb2c48a-65ce-436f-8e39-29d5f7aa0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 20:16:47.628712 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:47.628273 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r67h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:16:47.628712 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:47.628347 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" podUID="7fb2c48a-65ce-436f-8e39-29d5f7aa0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 20:16:48.625252 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:48.625211 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r67h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:16:48.625438 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:48.625279 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" podUID="7fb2c48a-65ce-436f-8e39-29d5f7aa0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 20:16:50.298841 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:50.298806 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:50.299362 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:50.299340 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:50.300146 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:16:50.300125 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" Oct 31 20:16:50.402739 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:16:50.402705 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:17:20.403085 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:17:20.403002 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:17:50.403188 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:17:50.403143 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:18:20.409947 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:18:20.409858 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:18:50.403365 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:18:50.403284 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:19:20.403724 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:19:20.403686 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:19:50.403150 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:19:50.403102 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:20:20.484514 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:20:20.484419 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:20:50.401959 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:20:50.401922 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:21:20.403250 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:21:20.403208 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:21:50.404787 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:21:50.404689 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:21:53.643556 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.643518 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk"] Oct 31 20:21:53.647112 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.647087 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" Oct 31 20:21:53.697967 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.697847 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b50005\"" Oct 31 20:21:53.745838 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.745798 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk"] Oct 31 20:21:53.764239 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.764205 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5964da-9a03-40ab-848c-765e3900d59b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk\" (UID: \"8e5964da-9a03-40ab-848c-765e3900d59b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" Oct 31 20:21:53.764440 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.764264 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk46q\" (UniqueName: \"kubernetes.io/projected/8e5964da-9a03-40ab-848c-765e3900d59b-kube-api-access-qk46q\") pod \"payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk\" (UID: \"8e5964da-9a03-40ab-848c-765e3900d59b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" Oct 31 20:21:53.784817 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.784780 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k"] Oct 31 20:21:53.788238 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.788217 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" Oct 31 20:21:53.840974 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.840939 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-99a6788f-dockercfg-sfj7c\"" Oct 31 20:21:53.848911 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.848794 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k"] Oct 31 20:21:53.865791 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.865743 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk46q\" (UniqueName: \"kubernetes.io/projected/8e5964da-9a03-40ab-848c-765e3900d59b-kube-api-access-qk46q\") pod \"payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk\" (UID: \"8e5964da-9a03-40ab-848c-765e3900d59b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" Oct 31 20:21:53.865791 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.865804 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxkcz\" (UniqueName: \"kubernetes.io/projected/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-kube-api-access-wxkcz\") pod \"payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k\" (UID: \"4ad1c90c-829e-40b6-aa14-58ed9d84ee86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" Oct 31 20:21:53.866047 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.865984 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k\" (UID: \"4ad1c90c-829e-40b6-aa14-58ed9d84ee86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" Oct 31 20:21:53.866134 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.866105 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5964da-9a03-40ab-848c-765e3900d59b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk\" (UID: \"8e5964da-9a03-40ab-848c-765e3900d59b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" Oct 31 20:21:53.866848 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.866830 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5964da-9a03-40ab-848c-765e3900d59b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk\" (UID: \"8e5964da-9a03-40ab-848c-765e3900d59b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" Oct 31 20:21:53.918773 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.918686 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk46q\" (UniqueName: \"kubernetes.io/projected/8e5964da-9a03-40ab-848c-765e3900d59b-kube-api-access-qk46q\") pod \"payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk\" (UID: \"8e5964da-9a03-40ab-848c-765e3900d59b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" Oct 31 20:21:53.959907 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.959858 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" Oct 31 20:21:53.967246 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.967216 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxkcz\" (UniqueName: \"kubernetes.io/projected/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-kube-api-access-wxkcz\") pod \"payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k\" (UID: \"4ad1c90c-829e-40b6-aa14-58ed9d84ee86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" Oct 31 20:21:53.967393 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.967282 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k\" (UID: \"4ad1c90c-829e-40b6-aa14-58ed9d84ee86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" Oct 31 20:21:53.968000 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:53.967978 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k\" (UID: \"4ad1c90c-829e-40b6-aa14-58ed9d84ee86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" Oct 31 20:21:54.020873 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:54.020804 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxkcz\" (UniqueName: \"kubernetes.io/projected/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-kube-api-access-wxkcz\") pod \"payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k\" (UID: \"4ad1c90c-829e-40b6-aa14-58ed9d84ee86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" Oct 31 20:21:54.101504 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:54.101459 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" Oct 31 20:21:54.321859 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:54.321829 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk"] Oct 31 20:21:54.324708 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:21:54.324680 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5964da_9a03_40ab_848c_765e3900d59b.slice/crio-c8657034607460a2150ec93335b85756e9a82144414b2f92fa3e2f1cbbc50259 WatchSource:0}: Error finding container c8657034607460a2150ec93335b85756e9a82144414b2f92fa3e2f1cbbc50259: Status 404 returned error can't find the container with id c8657034607460a2150ec93335b85756e9a82144414b2f92fa3e2f1cbbc50259 Oct 31 20:21:54.485618 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:54.485594 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k"] Oct 31 20:21:54.486842 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:21:54.486812 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad1c90c_829e_40b6_aa14_58ed9d84ee86.slice/crio-115b2f47ea3e27bd01f77b204136a4c3b5699ed29da635c23ddf60ec6b72e3a7 WatchSource:0}: Error finding container 115b2f47ea3e27bd01f77b204136a4c3b5699ed29da635c23ddf60ec6b72e3a7: Status 404 returned error can't find the container with id 115b2f47ea3e27bd01f77b204136a4c3b5699ed29da635c23ddf60ec6b72e3a7 Oct 31 20:21:55.051052 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:55.051009 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" event={"ID":"8e5964da-9a03-40ab-848c-765e3900d59b","Type":"ContainerStarted","Data":"baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa"} Oct 31 20:21:55.051541 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:55.051167 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" Oct 31 20:21:55.051541 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:55.051216 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" event={"ID":"8e5964da-9a03-40ab-848c-765e3900d59b","Type":"ContainerStarted","Data":"c8657034607460a2150ec93335b85756e9a82144414b2f92fa3e2f1cbbc50259"} Oct 31 20:21:55.052558 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:55.052540 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" event={"ID":"4ad1c90c-829e-40b6-aa14-58ed9d84ee86","Type":"ContainerStarted","Data":"99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc"} Oct 31 20:21:55.052625 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:55.052574 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" event={"ID":"4ad1c90c-829e-40b6-aa14-58ed9d84ee86","Type":"ContainerStarted","Data":"115b2f47ea3e27bd01f77b204136a4c3b5699ed29da635c23ddf60ec6b72e3a7"} Oct 31 20:21:55.052625 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:55.052598 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" Oct 31 20:21:55.157368 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:55.157307 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" podStartSLOduration=2.157291665 podStartE2EDuration="2.157291665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:21:55.155354254 +0000 UTC m=+3886.788748161" watchObservedRunningTime="2026-10-31 20:21:55.157291665 +0000 UTC m=+3886.790685570" Oct 31 20:21:55.261970 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:21:55.261908 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" podStartSLOduration=2.261867568 podStartE2EDuration="2.261867568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:21:55.261186023 +0000 UTC m=+3886.894579929" watchObservedRunningTime="2026-10-31 20:21:55.261867568 +0000 UTC m=+3886.895261523" Oct 31 20:22:06.060753 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:06.060724 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" Oct 31 20:22:06.061165 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:06.060797 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" Oct 31 20:22:17.555890 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.555849 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7"] Oct 31 20:22:17.560163 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.560138 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.611751 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.611713 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-home\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.611941 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.611830 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-model-cache\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.611996 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.611946 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-dshm\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.612044 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.612014 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-tmp-dir\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.612102 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.612083 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4754f793-c8ae-4334-8e4b-db985266c4fa-tls-certs\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.612186 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.612166 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69shl\" (UniqueName: \"kubernetes.io/projected/4754f793-c8ae-4334-8e4b-db985266c4fa-kube-api-access-69shl\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.655028 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.654998 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b50005\"/\"kube-root-ca.crt\"" Oct 31 20:22:17.655208 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.655039 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b50005\"/\"default-dockercfg-qtr7t\"" Oct 31 20:22:17.655208 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.655058 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b50005\"/\"openshift-service-ca.crt\"" Oct 31 20:22:17.655208 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.655042 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b50005\"/\"simulator-single-main-b50005-kserve-self-signed-certs\"" Oct 31 20:22:17.657098 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.657078 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7"] Oct 31 20:22:17.713508 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.713476 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-dshm\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.713740 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.713537 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-tmp-dir\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.713740 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.713611 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4754f793-c8ae-4334-8e4b-db985266c4fa-tls-certs\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.713740 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.713654 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69shl\" (UniqueName: \"kubernetes.io/projected/4754f793-c8ae-4334-8e4b-db985266c4fa-kube-api-access-69shl\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.713740 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.713691 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-home\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.714027 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.713753 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-model-cache\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.714100 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.714077 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-tmp-dir\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.714158 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.714141 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-home\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.714213 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.714167 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-model-cache\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.715956 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.715913 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-dshm\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.716822 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.716795 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4754f793-c8ae-4334-8e4b-db985266c4fa-tls-certs\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.766521 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.766488 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69shl\" (UniqueName: \"kubernetes.io/projected/4754f793-c8ae-4334-8e4b-db985266c4fa-kube-api-access-69shl\") pod \"simulator-single-main-b50005-kserve-8588b6b968-6t5r7\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:17.872857 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:17.872774 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:18.237753 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:18.237557 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7"] Oct 31 20:22:18.240845 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:22:18.240816 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4754f793_c8ae_4334_8e4b_db985266c4fa.slice/crio-e8420c4ef779f06ac8aa8e194cf1782f1aa13528193b9a01429cc97fac53c872 WatchSource:0}: Error finding container e8420c4ef779f06ac8aa8e194cf1782f1aa13528193b9a01429cc97fac53c872: Status 404 returned error can't find the container with id e8420c4ef779f06ac8aa8e194cf1782f1aa13528193b9a01429cc97fac53c872 Oct 31 20:22:19.180751 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:19.180704 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" event={"ID":"4754f793-c8ae-4334-8e4b-db985266c4fa","Type":"ContainerStarted","Data":"acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84"} Oct 31 20:22:19.181236 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:19.180758 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" event={"ID":"4754f793-c8ae-4334-8e4b-db985266c4fa","Type":"ContainerStarted","Data":"e8420c4ef779f06ac8aa8e194cf1782f1aa13528193b9a01429cc97fac53c872"} Oct 31 20:22:19.288018 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:19.287957 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" podStartSLOduration=2.287943134 podStartE2EDuration="2.287943134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:22:19.286148485 +0000 UTC m=+3910.919542398" watchObservedRunningTime="2026-10-31 20:22:19.287943134 +0000 UTC m=+3910.921337042" Oct 31 20:22:19.873681 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:19.873649 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:20.403431 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:22:20.403395 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:22:27.873038 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:27.872996 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:27.885913 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:27.885865 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:28.236376 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:22:28.236346 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:22:50.403205 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:22:50.403169 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:23:19.026160 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:19.026080 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7"] Oct 31 20:23:20.402368 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:23:20.402328 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:23:26.106043 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:26.106006 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k"] Oct 31 20:23:26.233813 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:26.233771 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk"] Oct 31 20:23:31.106362 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:31.106281 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" podUID="4ad1c90c-829e-40b6-aa14-58ed9d84ee86" containerName="payload-processing" containerID="cri-o://99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc" gracePeriod=25 Oct 31 20:23:31.235286 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:31.235239 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" podUID="8e5964da-9a03-40ab-848c-765e3900d59b" containerName="payload-pre-processing" containerID="cri-o://baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa" gracePeriod=25 Oct 31 20:23:34.046049 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.045990 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" podUID="4754f793-c8ae-4334-8e4b-db985266c4fa" containerName="main" containerID="cri-o://acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84" gracePeriod=45 Oct 31 20:23:34.296382 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.296325 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:23:34.378070 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.378036 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-home\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-home\") pod \"4754f793-c8ae-4334-8e4b-db985266c4fa\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " Oct 31 20:23:34.378265 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.378093 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-model-cache\") pod \"4754f793-c8ae-4334-8e4b-db985266c4fa\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " Oct 31 20:23:34.378265 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.378126 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-tmp-dir\") pod \"4754f793-c8ae-4334-8e4b-db985266c4fa\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " Oct 31 20:23:34.378265 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.378150 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4754f793-c8ae-4334-8e4b-db985266c4fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/4754f793-c8ae-4334-8e4b-db985266c4fa-tls-certs\") pod \"4754f793-c8ae-4334-8e4b-db985266c4fa\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " Oct 31 20:23:34.378265 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.378178 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4754f793-c8ae-4334-8e4b-db985266c4fa-kube-api-access-69shl\" (UniqueName: \"kubernetes.io/projected/4754f793-c8ae-4334-8e4b-db985266c4fa-kube-api-access-69shl\") pod \"4754f793-c8ae-4334-8e4b-db985266c4fa\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " Oct 31 20:23:34.378265 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.378203 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-dshm\") pod \"4754f793-c8ae-4334-8e4b-db985266c4fa\" (UID: \"4754f793-c8ae-4334-8e4b-db985266c4fa\") " Oct 31 20:23:34.378521 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.378353 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-home" pod "4754f793-c8ae-4334-8e4b-db985266c4fa" (UID: "4754f793-c8ae-4334-8e4b-db985266c4fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:34.378521 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.378413 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-model-cache" pod "4754f793-c8ae-4334-8e4b-db985266c4fa" (UID: "4754f793-c8ae-4334-8e4b-db985266c4fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:34.378521 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.378419 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-tmp-dir" pod "4754f793-c8ae-4334-8e4b-db985266c4fa" (UID: "4754f793-c8ae-4334-8e4b-db985266c4fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:34.378635 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.378618 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:34.378683 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.378639 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:34.378683 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.378654 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:34.380448 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.380425 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4754f793-c8ae-4334-8e4b-db985266c4fa-tls-certs" pod "4754f793-c8ae-4334-8e4b-db985266c4fa" (UID: "4754f793-c8ae-4334-8e4b-db985266c4fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:23:34.380529 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.380493 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4754f793-c8ae-4334-8e4b-db985266c4fa-kube-api-access-69shl" pod "4754f793-c8ae-4334-8e4b-db985266c4fa" (UID: "4754f793-c8ae-4334-8e4b-db985266c4fa"). InnerVolumeSpecName "kube-api-access-69shl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:23:34.380529 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.380516 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-dshm" pod "4754f793-c8ae-4334-8e4b-db985266c4fa" (UID: "4754f793-c8ae-4334-8e4b-db985266c4fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:34.479811 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.479774 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4754f793-c8ae-4334-8e4b-db985266c4fa-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:34.480095 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.480049 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69shl\" (UniqueName: \"kubernetes.io/projected/4754f793-c8ae-4334-8e4b-db985266c4fa-kube-api-access-69shl\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:34.480095 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.480097 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4754f793-c8ae-4334-8e4b-db985266c4fa-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:34.579178 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.579090 6107 generic.go:356] "Generic (PLEG): container finished" podID="4754f793-c8ae-4334-8e4b-db985266c4fa" containerID="acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84" exitCode=0 Oct 31 20:23:34.579178 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.579136 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" Oct 31 20:23:34.579382 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.579179 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" event={"ID":"4754f793-c8ae-4334-8e4b-db985266c4fa","Type":"ContainerDied","Data":"acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84"} Oct 31 20:23:34.579382 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.579222 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" event={"ID":"4754f793-c8ae-4334-8e4b-db985266c4fa","Type":"ContainerDied","Data":"e8420c4ef779f06ac8aa8e194cf1782f1aa13528193b9a01429cc97fac53c872"} Oct 31 20:23:34.579382 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.579239 6107 scope.go:122] "RemoveContainer" containerID="acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84" Oct 31 20:23:34.588095 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.588075 6107 scope.go:122] "RemoveContainer" containerID="acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84" Oct 31 20:23:34.588357 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:23:34.588339 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84\": container with ID starting with acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84 not found: ID does not exist" containerID="acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84" Oct 31 20:23:34.588405 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.588366 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84"} err="failed to get container status \"acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84\": rpc error: code = NotFound desc = could not find container \"acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84\": container with ID starting with acabd81361769acd0b52a945957abca9561f2bc25b2d0113b16d78e03dac0b84 not found: ID does not exist" Oct 31 20:23:34.729483 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.729452 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7"] Oct 31 20:23:34.736224 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:34.736198 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7"] Oct 31 20:23:35.227034 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:35.226995 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-b50005/simulator-single-main-b50005-kserve-8588b6b968-6t5r7" podUID="4754f793-c8ae-4334-8e4b-db985266c4fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:23:36.059746 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:36.059707 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" podUID="8e5964da-9a03-40ab-848c-765e3900d59b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:23:36.059949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:36.059707 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" podUID="4ad1c90c-829e-40b6-aa14-58ed9d84ee86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:23:36.486002 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:36.485963 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4754f793-c8ae-4334-8e4b-db985266c4fa" path="/var/lib/kubelet/pods/4754f793-c8ae-4334-8e4b-db985266c4fa/volumes" Oct 31 20:23:46.059222 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:46.059180 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" podUID="4ad1c90c-829e-40b6-aa14-58ed9d84ee86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:23:46.059222 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:46.059218 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" podUID="8e5964da-9a03-40ab-848c-765e3900d59b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:23:50.403760 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:23:50.403724 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:23:51.356669 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.356647 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" Oct 31 20:23:51.443398 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.443370 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-kube-api-access-wxkcz\" (UniqueName: \"kubernetes.io/projected/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-kube-api-access-wxkcz\") pod \"4ad1c90c-829e-40b6-aa14-58ed9d84ee86\" (UID: \"4ad1c90c-829e-40b6-aa14-58ed9d84ee86\") " Oct 31 20:23:51.443846 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.443478 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-plugins-config-volume\") pod \"4ad1c90c-829e-40b6-aa14-58ed9d84ee86\" (UID: \"4ad1c90c-829e-40b6-aa14-58ed9d84ee86\") " Oct 31 20:23:51.443946 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.443918 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-plugins-config-volume" pod "4ad1c90c-829e-40b6-aa14-58ed9d84ee86" (UID: "4ad1c90c-829e-40b6-aa14-58ed9d84ee86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:23:51.446412 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.446381 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-kube-api-access-wxkcz" pod "4ad1c90c-829e-40b6-aa14-58ed9d84ee86" (UID: "4ad1c90c-829e-40b6-aa14-58ed9d84ee86"). InnerVolumeSpecName "kube-api-access-wxkcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:23:51.467325 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.467303 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" Oct 31 20:23:51.544436 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.544405 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5964da-9a03-40ab-848c-765e3900d59b-kube-api-access-qk46q\" (UniqueName: \"kubernetes.io/projected/8e5964da-9a03-40ab-848c-765e3900d59b-kube-api-access-qk46q\") pod \"8e5964da-9a03-40ab-848c-765e3900d59b\" (UID: \"8e5964da-9a03-40ab-848c-765e3900d59b\") " Oct 31 20:23:51.544603 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.544453 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5964da-9a03-40ab-848c-765e3900d59b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5964da-9a03-40ab-848c-765e3900d59b-plugins-config-volume\") pod \"8e5964da-9a03-40ab-848c-765e3900d59b\" (UID: \"8e5964da-9a03-40ab-848c-765e3900d59b\") " Oct 31 20:23:51.544731 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.544710 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:51.544776 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.544738 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxkcz\" (UniqueName: \"kubernetes.io/projected/4ad1c90c-829e-40b6-aa14-58ed9d84ee86-kube-api-access-wxkcz\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:51.544827 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.544812 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5964da-9a03-40ab-848c-765e3900d59b-plugins-config-volume" pod "8e5964da-9a03-40ab-848c-765e3900d59b" (UID: "8e5964da-9a03-40ab-848c-765e3900d59b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:23:51.546583 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.546562 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5964da-9a03-40ab-848c-765e3900d59b-kube-api-access-qk46q" pod "8e5964da-9a03-40ab-848c-765e3900d59b" (UID: "8e5964da-9a03-40ab-848c-765e3900d59b"). InnerVolumeSpecName "kube-api-access-qk46q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:23:51.645752 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.645719 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk46q\" (UniqueName: \"kubernetes.io/projected/8e5964da-9a03-40ab-848c-765e3900d59b-kube-api-access-qk46q\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:51.645752 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.645745 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5964da-9a03-40ab-848c-765e3900d59b-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:51.668269 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.668187 6107 generic.go:356] "Generic (PLEG): container finished" podID="4ad1c90c-829e-40b6-aa14-58ed9d84ee86" containerID="99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc" exitCode=0 Oct 31 20:23:51.668269 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.668230 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" Oct 31 20:23:51.668473 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.668269 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" event={"ID":"4ad1c90c-829e-40b6-aa14-58ed9d84ee86","Type":"ContainerDied","Data":"99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc"} Oct 31 20:23:51.668473 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.668319 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k" event={"ID":"4ad1c90c-829e-40b6-aa14-58ed9d84ee86","Type":"ContainerDied","Data":"115b2f47ea3e27bd01f77b204136a4c3b5699ed29da635c23ddf60ec6b72e3a7"} Oct 31 20:23:51.668473 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.668332 6107 scope.go:122] "RemoveContainer" containerID="99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc" Oct 31 20:23:51.669821 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.669800 6107 generic.go:356] "Generic (PLEG): container finished" podID="8e5964da-9a03-40ab-848c-765e3900d59b" containerID="baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa" exitCode=0 Oct 31 20:23:51.669962 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.669831 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" event={"ID":"8e5964da-9a03-40ab-848c-765e3900d59b","Type":"ContainerDied","Data":"baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa"} Oct 31 20:23:51.669962 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.669858 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" Oct 31 20:23:51.670059 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.669860 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk" event={"ID":"8e5964da-9a03-40ab-848c-765e3900d59b","Type":"ContainerDied","Data":"c8657034607460a2150ec93335b85756e9a82144414b2f92fa3e2f1cbbc50259"} Oct 31 20:23:51.678861 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.678846 6107 scope.go:122] "RemoveContainer" containerID="99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc" Oct 31 20:23:51.679194 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:23:51.679176 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc\": container with ID starting with 99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc not found: ID does not exist" containerID="99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc" Oct 31 20:23:51.679258 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.679202 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc"} err="failed to get container status \"99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc\": rpc error: code = NotFound desc = could not find container \"99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc\": container with ID starting with 99fbd07048a121b59cf2614f9b92f7208511eb5af74533ff23d6d3672f4ac3dc not found: ID does not exist" Oct 31 20:23:51.679258 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.679221 6107 scope.go:122] "RemoveContainer" containerID="baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa" Oct 31 20:23:51.687726 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.687709 6107 scope.go:122] "RemoveContainer" containerID="baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa" Oct 31 20:23:51.688069 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:23:51.688052 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa\": container with ID starting with baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa not found: ID does not exist" containerID="baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa" Oct 31 20:23:51.688139 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.688075 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa"} err="failed to get container status \"baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa\": rpc error: code = NotFound desc = could not find container \"baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa\": container with ID starting with baa8c61270320bf300b09c764dc58ade61ce203a3496b077c471adef664644fa not found: ID does not exist" Oct 31 20:23:51.819692 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.819658 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk"] Oct 31 20:23:51.823560 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.823534 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b50005-58bbbtbwwk"] Oct 31 20:23:51.924856 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.924772 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k"] Oct 31 20:23:51.928430 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:51.928402 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b50005-7bb57bc9c5rb7k"] Oct 31 20:23:52.485246 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:52.485215 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad1c90c-829e-40b6-aa14-58ed9d84ee86" path="/var/lib/kubelet/pods/4ad1c90c-829e-40b6-aa14-58ed9d84ee86/volumes" Oct 31 20:23:52.485621 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:23:52.485598 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5964da-9a03-40ab-848c-765e3900d59b" path="/var/lib/kubelet/pods/8e5964da-9a03-40ab-848c-765e3900d59b/volumes" Oct 31 20:24:03.446378 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.446344 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv"] Oct 31 20:24:03.446942 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.446547 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" podUID="19ec27d2-0211-40fa-a856-4a95d99c5209" containerName="manager" containerID="cri-o://7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717" gracePeriod=10 Oct 31 20:24:03.696504 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.696449 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" Oct 31 20:24:03.738914 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.738867 6107 generic.go:356] "Generic (PLEG): container finished" podID="19ec27d2-0211-40fa-a856-4a95d99c5209" containerID="7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717" exitCode=0 Oct 31 20:24:03.739074 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.738925 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" Oct 31 20:24:03.739074 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.738918 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" event={"ID":"19ec27d2-0211-40fa-a856-4a95d99c5209","Type":"ContainerDied","Data":"7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717"} Oct 31 20:24:03.739074 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.739040 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv" event={"ID":"19ec27d2-0211-40fa-a856-4a95d99c5209","Type":"ContainerDied","Data":"6df1253d3aa3da1561ea02989a4f88f757181c9dc157132669e5bbdbab6a27b5"} Oct 31 20:24:03.739074 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.739067 6107 scope.go:122] "RemoveContainer" containerID="7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717" Oct 31 20:24:03.751520 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.751501 6107 scope.go:122] "RemoveContainer" containerID="7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717" Oct 31 20:24:03.751791 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:24:03.751765 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717\": container with ID starting with 7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717 not found: ID does not exist" containerID="7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717" Oct 31 20:24:03.751866 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.751804 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717"} err="failed to get container status \"7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717\": rpc error: code = NotFound desc = could not find container \"7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717\": container with ID starting with 7e19a1c590d0b7bcca9cea8bc30bb794a244be8f8a6c82f20995c555d4a59717 not found: ID does not exist" Oct 31 20:24:03.872032 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.872002 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ec27d2-0211-40fa-a856-4a95d99c5209-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19ec27d2-0211-40fa-a856-4a95d99c5209-extensions-socket-volume\") pod \"19ec27d2-0211-40fa-a856-4a95d99c5209\" (UID: \"19ec27d2-0211-40fa-a856-4a95d99c5209\") " Oct 31 20:24:03.872226 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.872089 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ec27d2-0211-40fa-a856-4a95d99c5209-kube-api-access-chf22\" (UniqueName: \"kubernetes.io/projected/19ec27d2-0211-40fa-a856-4a95d99c5209-kube-api-access-chf22\") pod \"19ec27d2-0211-40fa-a856-4a95d99c5209\" (UID: \"19ec27d2-0211-40fa-a856-4a95d99c5209\") " Oct 31 20:24:03.872423 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.872400 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ec27d2-0211-40fa-a856-4a95d99c5209-extensions-socket-volume" pod "19ec27d2-0211-40fa-a856-4a95d99c5209" (UID: "19ec27d2-0211-40fa-a856-4a95d99c5209"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:24:03.874112 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.874088 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ec27d2-0211-40fa-a856-4a95d99c5209-kube-api-access-chf22" pod "19ec27d2-0211-40fa-a856-4a95d99c5209" (UID: "19ec27d2-0211-40fa-a856-4a95d99c5209"). InnerVolumeSpecName "kube-api-access-chf22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:24:03.973203 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.973170 6107 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19ec27d2-0211-40fa-a856-4a95d99c5209-extensions-socket-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:24:03.973203 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:03.973200 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chf22\" (UniqueName: \"kubernetes.io/projected/19ec27d2-0211-40fa-a856-4a95d99c5209-kube-api-access-chf22\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:24:04.151182 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:04.151145 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv"] Oct 31 20:24:04.152989 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:04.152962 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fvzlv"] Oct 31 20:24:04.485177 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:24:04.485144 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ec27d2-0211-40fa-a856-4a95d99c5209" path="/var/lib/kubelet/pods/19ec27d2-0211-40fa-a856-4a95d99c5209/volumes" Oct 31 20:24:20.402838 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:24:20.402796 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:24:50.402864 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:24:50.402765 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:25:06.995412 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995373 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r"] Oct 31 20:25:06.995844 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995765 6107 cpu_manager.go:414] "containerMap: removing container" podUID="4ad1c90c-829e-40b6-aa14-58ed9d84ee86" containerName="payload-processing" Oct 31 20:25:06.995844 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995776 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad1c90c-829e-40b6-aa14-58ed9d84ee86" containerName="payload-processing" Oct 31 20:25:06.995844 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995788 6107 cpu_manager.go:414] "containerMap: removing container" podUID="19ec27d2-0211-40fa-a856-4a95d99c5209" containerName="manager" Oct 31 20:25:06.995844 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995793 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ec27d2-0211-40fa-a856-4a95d99c5209" containerName="manager" Oct 31 20:25:06.995844 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995804 6107 cpu_manager.go:414] "containerMap: removing container" podUID="8e5964da-9a03-40ab-848c-765e3900d59b" containerName="payload-pre-processing" Oct 31 20:25:06.995844 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995808 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5964da-9a03-40ab-848c-765e3900d59b" containerName="payload-pre-processing" Oct 31 20:25:06.995844 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995815 6107 cpu_manager.go:414] "containerMap: removing container" podUID="4754f793-c8ae-4334-8e4b-db985266c4fa" containerName="main" Oct 31 20:25:06.995844 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995820 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4754f793-c8ae-4334-8e4b-db985266c4fa" containerName="main" Oct 31 20:25:06.996157 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995906 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad1c90c-829e-40b6-aa14-58ed9d84ee86" containerName="payload-processing" Oct 31 20:25:06.996157 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995914 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ec27d2-0211-40fa-a856-4a95d99c5209" containerName="manager" Oct 31 20:25:06.996157 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995922 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5964da-9a03-40ab-848c-765e3900d59b" containerName="payload-pre-processing" Oct 31 20:25:06.996157 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.995927 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="4754f793-c8ae-4334-8e4b-db985266c4fa" containerName="main" Oct 31 20:25:06.999165 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:06.999149 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" Oct 31 20:25:07.047601 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:07.047569 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 20:25:07.076520 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:07.076487 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ad70bd3-5d6a-4631-b5fe-e555d708bb44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-89l6r\" (UID: \"7ad70bd3-5d6a-4631-b5fe-e555d708bb44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" Oct 31 20:25:07.076520 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:07.076522 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxj4m\" (UniqueName: \"kubernetes.io/projected/7ad70bd3-5d6a-4631-b5fe-e555d708bb44-kube-api-access-zxj4m\") pod \"kuadrant-operator-controller-manager-678d87766f-89l6r\" (UID: \"7ad70bd3-5d6a-4631-b5fe-e555d708bb44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" Oct 31 20:25:07.089877 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:07.089846 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vkv8v\"" Oct 31 20:25:07.090049 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:07.089966 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 20:25:07.094964 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:07.094939 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r"] Oct 31 20:25:07.178068 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:07.178039 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ad70bd3-5d6a-4631-b5fe-e555d708bb44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-89l6r\" (UID: \"7ad70bd3-5d6a-4631-b5fe-e555d708bb44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" Oct 31 20:25:07.178234 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:07.178076 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxj4m\" (UniqueName: \"kubernetes.io/projected/7ad70bd3-5d6a-4631-b5fe-e555d708bb44-kube-api-access-zxj4m\") pod \"kuadrant-operator-controller-manager-678d87766f-89l6r\" (UID: \"7ad70bd3-5d6a-4631-b5fe-e555d708bb44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" Oct 31 20:25:07.178423 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:07.178404 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ad70bd3-5d6a-4631-b5fe-e555d708bb44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-89l6r\" (UID: \"7ad70bd3-5d6a-4631-b5fe-e555d708bb44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" Oct 31 20:25:07.231336 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:07.231308 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxj4m\" (UniqueName: \"kubernetes.io/projected/7ad70bd3-5d6a-4631-b5fe-e555d708bb44-kube-api-access-zxj4m\") pod \"kuadrant-operator-controller-manager-678d87766f-89l6r\" (UID: \"7ad70bd3-5d6a-4631-b5fe-e555d708bb44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" Oct 31 20:25:07.312070 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:07.312001 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" Oct 31 20:25:07.723443 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:07.723420 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r"] Oct 31 20:25:07.725745 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:25:07.725712 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad70bd3_5d6a_4631_b5fe_e555d708bb44.slice/crio-8c339b8ff6068289c43be351dd1d783ea8cab84e1a178352c61b4048f19fa16e WatchSource:0}: Error finding container 8c339b8ff6068289c43be351dd1d783ea8cab84e1a178352c61b4048f19fa16e: Status 404 returned error can't find the container with id 8c339b8ff6068289c43be351dd1d783ea8cab84e1a178352c61b4048f19fa16e Oct 31 20:25:08.055442 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:08.055356 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" event={"ID":"7ad70bd3-5d6a-4631-b5fe-e555d708bb44","Type":"ContainerStarted","Data":"fe5b45a8cb0b84ed3211d0db64700a164ab2449b797c455da93dccc0b14fd562"} Oct 31 20:25:08.055442 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:08.055416 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" Oct 31 20:25:08.055442 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:08.055439 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" event={"ID":"7ad70bd3-5d6a-4631-b5fe-e555d708bb44","Type":"ContainerStarted","Data":"8c339b8ff6068289c43be351dd1d783ea8cab84e1a178352c61b4048f19fa16e"} Oct 31 20:25:08.164808 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:08.164749 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" podStartSLOduration=2.164733694 podStartE2EDuration="2.164733694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:25:08.162864709 +0000 UTC m=+4079.796258617" watchObservedRunningTime="2026-10-31 20:25:08.164733694 +0000 UTC m=+4079.798127601" Oct 31 20:25:19.062821 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:19.062787 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89l6r" Oct 31 20:25:20.402840 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:25:20.402794 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:25:50.460124 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:25:50.460044 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:25:51.349083 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:51.349041 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r67h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:25:51.349443 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:25:51.349392 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r67h6" podUID="7fb2c48a-65ce-436f-8e39-29d5f7aa0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:26:20.406066 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:26:20.405950 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:26:21.024448 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.024412 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn"] Oct 31 20:26:21.027840 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.027819 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" Oct 31 20:26:21.082099 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.082068 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn"] Oct 31 20:26:21.125622 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.125588 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d28b6b\"" Oct 31 20:26:21.125945 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.125763 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e7ea0dbd-dockercfg-4pqpm\"" Oct 31 20:26:21.162410 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.162366 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f213fa-a773-4d24-a5f9-b9a66b76b449-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn\" (UID: \"27f213fa-a773-4d24-a5f9-b9a66b76b449\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" Oct 31 20:26:21.162584 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.162494 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ncc\" (UniqueName: \"kubernetes.io/projected/27f213fa-a773-4d24-a5f9-b9a66b76b449-kube-api-access-q2ncc\") pod \"payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn\" (UID: \"27f213fa-a773-4d24-a5f9-b9a66b76b449\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" Oct 31 20:26:21.173452 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.173423 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b"] Oct 31 20:26:21.177493 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.177471 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" Oct 31 20:26:21.220128 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.220095 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b"] Oct 31 20:26:21.264100 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.264063 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b\" (UID: \"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" Oct 31 20:26:21.264270 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.264106 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f213fa-a773-4d24-a5f9-b9a66b76b449-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn\" (UID: \"27f213fa-a773-4d24-a5f9-b9a66b76b449\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" Oct 31 20:26:21.264270 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.264206 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24mlm\" (UniqueName: \"kubernetes.io/projected/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-kube-api-access-24mlm\") pod \"payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b\" (UID: \"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" Oct 31 20:26:21.264270 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.264244 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ncc\" (UniqueName: \"kubernetes.io/projected/27f213fa-a773-4d24-a5f9-b9a66b76b449-kube-api-access-q2ncc\") pod \"payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn\" (UID: \"27f213fa-a773-4d24-a5f9-b9a66b76b449\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" Oct 31 20:26:21.264816 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.264800 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f213fa-a773-4d24-a5f9-b9a66b76b449-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn\" (UID: \"27f213fa-a773-4d24-a5f9-b9a66b76b449\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" Oct 31 20:26:21.317496 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.317412 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ncc\" (UniqueName: \"kubernetes.io/projected/27f213fa-a773-4d24-a5f9-b9a66b76b449-kube-api-access-q2ncc\") pod \"payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn\" (UID: \"27f213fa-a773-4d24-a5f9-b9a66b76b449\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" Oct 31 20:26:21.340316 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.340280 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" Oct 31 20:26:21.365746 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.365717 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24mlm\" (UniqueName: \"kubernetes.io/projected/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-kube-api-access-24mlm\") pod \"payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b\" (UID: \"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" Oct 31 20:26:21.365892 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.365845 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b\" (UID: \"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" Oct 31 20:26:21.366545 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.366525 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b\" (UID: \"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" Oct 31 20:26:21.419840 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.419806 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24mlm\" (UniqueName: \"kubernetes.io/projected/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-kube-api-access-24mlm\") pod \"payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b\" (UID: \"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" Oct 31 20:26:21.491838 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.491805 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" Oct 31 20:26:21.703186 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.703145 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn"] Oct 31 20:26:21.705495 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:26:21.705466 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f213fa_a773_4d24_a5f9_b9a66b76b449.slice/crio-26b09f9c783b0db3a751433bfaa470efd6022819afd9e9b4bf08368b5fa3088f WatchSource:0}: Error finding container 26b09f9c783b0db3a751433bfaa470efd6022819afd9e9b4bf08368b5fa3088f: Status 404 returned error can't find the container with id 26b09f9c783b0db3a751433bfaa470efd6022819afd9e9b4bf08368b5fa3088f Oct 31 20:26:21.855057 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:21.855030 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b"] Oct 31 20:26:21.857276 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:26:21.857245 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b6a8f2_b3ea_4b35_83ca_25fa491e1b30.slice/crio-cbca53c3ba90b268ca9561397da31c5798c711ab9eec74b6af082057e41a1c9e WatchSource:0}: Error finding container cbca53c3ba90b268ca9561397da31c5798c711ab9eec74b6af082057e41a1c9e: Status 404 returned error can't find the container with id cbca53c3ba90b268ca9561397da31c5798c711ab9eec74b6af082057e41a1c9e Oct 31 20:26:22.447824 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:22.447784 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" event={"ID":"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30","Type":"ContainerStarted","Data":"68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2"} Oct 31 20:26:22.448358 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:22.447848 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" event={"ID":"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30","Type":"ContainerStarted","Data":"cbca53c3ba90b268ca9561397da31c5798c711ab9eec74b6af082057e41a1c9e"} Oct 31 20:26:22.448358 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:22.447954 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" Oct 31 20:26:22.449395 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:22.449370 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" event={"ID":"27f213fa-a773-4d24-a5f9-b9a66b76b449","Type":"ContainerStarted","Data":"96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f"} Oct 31 20:26:22.449525 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:22.449413 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" event={"ID":"27f213fa-a773-4d24-a5f9-b9a66b76b449","Type":"ContainerStarted","Data":"26b09f9c783b0db3a751433bfaa470efd6022819afd9e9b4bf08368b5fa3088f"} Oct 31 20:26:22.449525 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:22.449452 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" Oct 31 20:26:22.563369 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:22.563300 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" podStartSLOduration=1.563282231 podStartE2EDuration="1.563282231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:26:22.562925508 +0000 UTC m=+4154.196319417" watchObservedRunningTime="2026-10-31 20:26:22.563282231 +0000 UTC m=+4154.196676146" Oct 31 20:26:22.670526 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:22.670466 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" podStartSLOduration=2.670450432 podStartE2EDuration="2.670450432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:26:22.669051846 +0000 UTC m=+4154.302445754" watchObservedRunningTime="2026-10-31 20:26:22.670450432 +0000 UTC m=+4154.303844336" Oct 31 20:26:33.457827 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:33.457794 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" Oct 31 20:26:33.458258 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:33.457871 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" Oct 31 20:26:46.222065 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.222028 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv"] Oct 31 20:26:46.225475 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.225452 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.317168 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.317131 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0117a9-5192-4dfd-b305-b280a65255c6-tls-certs\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.317346 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.317190 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn2w6\" (UniqueName: \"kubernetes.io/projected/9f0117a9-5192-4dfd-b305-b280a65255c6-kube-api-access-cn2w6\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.317346 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.317318 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-dshm\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.317443 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.317430 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-tmp-dir\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.317516 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.317491 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-model-cache\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.317562 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.317529 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-home\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.322531 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.322513 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d28b6b\"/\"openshift-service-ca.crt\"" Oct 31 20:26:46.322629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.322531 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d28b6b\"/\"simulator-single-main-d28b6b-kserve-self-signed-certs\"" Oct 31 20:26:46.322629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.322583 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d28b6b\"/\"kube-root-ca.crt\"" Oct 31 20:26:46.322773 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.322761 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d28b6b\"/\"default-dockercfg-r4wvm\"" Oct 31 20:26:46.324643 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.324625 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv"] Oct 31 20:26:46.418323 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.418290 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-dshm\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.418514 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.418405 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-tmp-dir\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.418514 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.418443 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-model-cache\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.418514 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.418471 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-home\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.418711 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.418519 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0117a9-5192-4dfd-b305-b280a65255c6-tls-certs\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.418711 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.418575 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2w6\" (UniqueName: \"kubernetes.io/projected/9f0117a9-5192-4dfd-b305-b280a65255c6-kube-api-access-cn2w6\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.418875 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.418846 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-tmp-dir\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.418984 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.418857 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-model-cache\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.418984 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.418917 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-home\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.421738 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.420654 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-dshm\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.421738 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.421126 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0117a9-5192-4dfd-b305-b280a65255c6-tls-certs\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.472830 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.472750 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn2w6\" (UniqueName: \"kubernetes.io/projected/9f0117a9-5192-4dfd-b305-b280a65255c6-kube-api-access-cn2w6\") pod \"simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.537813 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.537777 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:46.904470 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:46.904389 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv"] Oct 31 20:26:46.906872 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:26:46.906847 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f0117a9_5192_4dfd_b305_b280a65255c6.slice/crio-148d004d9ad36554eaa441e1812f4f3516a3900614f06cdda15ce76c6d73da1c WatchSource:0}: Error finding container 148d004d9ad36554eaa441e1812f4f3516a3900614f06cdda15ce76c6d73da1c: Status 404 returned error can't find the container with id 148d004d9ad36554eaa441e1812f4f3516a3900614f06cdda15ce76c6d73da1c Oct 31 20:26:47.582581 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:47.582540 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" event={"ID":"9f0117a9-5192-4dfd-b305-b280a65255c6","Type":"ContainerStarted","Data":"f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba"} Oct 31 20:26:47.582581 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:47.582585 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" event={"ID":"9f0117a9-5192-4dfd-b305-b280a65255c6","Type":"ContainerStarted","Data":"148d004d9ad36554eaa441e1812f4f3516a3900614f06cdda15ce76c6d73da1c"} Oct 31 20:26:47.690476 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:47.690422 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" podStartSLOduration=1.690408749 podStartE2EDuration="1.690408749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:26:47.689193321 +0000 UTC m=+4179.322587228" watchObservedRunningTime="2026-10-31 20:26:47.690408749 +0000 UTC m=+4179.323802656" Oct 31 20:26:48.538170 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:48.538124 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:50.403431 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:26:50.403393 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:26:56.538780 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:56.538743 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:56.551453 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:56.551425 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:26:56.647238 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:26:56.647203 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:27:20.403148 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:27:20.403106 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:27:46.459733 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:27:46.459697 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv"] Oct 31 20:27:50.402835 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:27:50.402757 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:27:53.518692 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:27:53.518662 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b"] Oct 31 20:27:53.626929 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:27:53.626893 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn"] Oct 31 20:27:58.519315 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:27:58.519258 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" podUID="b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30" containerName="payload-processing" containerID="cri-o://68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2" gracePeriod=25 Oct 31 20:27:58.627902 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:27:58.627839 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" podUID="27f213fa-a773-4d24-a5f9-b9a66b76b449" containerName="payload-pre-processing" containerID="cri-o://96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f" gracePeriod=25 Oct 31 20:28:01.479241 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.479185 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" podUID="9f0117a9-5192-4dfd-b305-b280a65255c6" containerName="main" containerID="cri-o://f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba" gracePeriod=45 Oct 31 20:28:01.636426 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.636380 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" podUID="9f0117a9-5192-4dfd-b305-b280a65255c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": dial tcp 10.133.0.139:8000: connect: connection refused" Oct 31 20:28:01.726525 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.726496 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:28:01.751265 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.751205 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-tmp-dir\") pod \"9f0117a9-5192-4dfd-b305-b280a65255c6\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " Oct 31 20:28:01.751388 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.751270 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-home\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-home\") pod \"9f0117a9-5192-4dfd-b305-b280a65255c6\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " Oct 31 20:28:01.751388 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.751324 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f0117a9-5192-4dfd-b305-b280a65255c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0117a9-5192-4dfd-b305-b280a65255c6-tls-certs\") pod \"9f0117a9-5192-4dfd-b305-b280a65255c6\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " Oct 31 20:28:01.751455 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.751391 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0117a9-5192-4dfd-b305-b280a65255c6-kube-api-access-cn2w6\" (UniqueName: \"kubernetes.io/projected/9f0117a9-5192-4dfd-b305-b280a65255c6-kube-api-access-cn2w6\") pod \"9f0117a9-5192-4dfd-b305-b280a65255c6\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " Oct 31 20:28:01.751455 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.751437 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-model-cache\") pod \"9f0117a9-5192-4dfd-b305-b280a65255c6\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " Oct 31 20:28:01.751527 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.751473 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-tmp-dir" pod "9f0117a9-5192-4dfd-b305-b280a65255c6" (UID: "9f0117a9-5192-4dfd-b305-b280a65255c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:28:01.751568 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.751523 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-dshm\") pod \"9f0117a9-5192-4dfd-b305-b280a65255c6\" (UID: \"9f0117a9-5192-4dfd-b305-b280a65255c6\") " Oct 31 20:28:01.751568 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.751543 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-home" pod "9f0117a9-5192-4dfd-b305-b280a65255c6" (UID: "9f0117a9-5192-4dfd-b305-b280a65255c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:28:01.751710 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.751692 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-model-cache" pod "9f0117a9-5192-4dfd-b305-b280a65255c6" (UID: "9f0117a9-5192-4dfd-b305-b280a65255c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:28:01.751862 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.751843 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:01.751942 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.751868 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:01.751942 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.751893 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:01.753636 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.753609 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0117a9-5192-4dfd-b305-b280a65255c6-kube-api-access-cn2w6" pod "9f0117a9-5192-4dfd-b305-b280a65255c6" (UID: "9f0117a9-5192-4dfd-b305-b280a65255c6"). InnerVolumeSpecName "kube-api-access-cn2w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:28:01.753743 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.753703 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0117a9-5192-4dfd-b305-b280a65255c6-tls-certs" pod "9f0117a9-5192-4dfd-b305-b280a65255c6" (UID: "9f0117a9-5192-4dfd-b305-b280a65255c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:28:01.753815 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.753780 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-dshm" pod "9f0117a9-5192-4dfd-b305-b280a65255c6" (UID: "9f0117a9-5192-4dfd-b305-b280a65255c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:28:01.852992 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.852958 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0117a9-5192-4dfd-b305-b280a65255c6-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:01.852992 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.852988 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0117a9-5192-4dfd-b305-b280a65255c6-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:01.853222 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.852999 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn2w6\" (UniqueName: \"kubernetes.io/projected/9f0117a9-5192-4dfd-b305-b280a65255c6-kube-api-access-cn2w6\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:01.978365 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.978328 6107 generic.go:356] "Generic (PLEG): container finished" podID="9f0117a9-5192-4dfd-b305-b280a65255c6" containerID="f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba" exitCode=0 Oct 31 20:28:01.978532 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.978373 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" event={"ID":"9f0117a9-5192-4dfd-b305-b280a65255c6","Type":"ContainerDied","Data":"f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba"} Oct 31 20:28:01.978532 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.978394 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" Oct 31 20:28:01.978532 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.978418 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv" event={"ID":"9f0117a9-5192-4dfd-b305-b280a65255c6","Type":"ContainerDied","Data":"148d004d9ad36554eaa441e1812f4f3516a3900614f06cdda15ce76c6d73da1c"} Oct 31 20:28:01.978532 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.978438 6107 scope.go:122] "RemoveContainer" containerID="f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba" Oct 31 20:28:01.992975 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.992946 6107 scope.go:122] "RemoveContainer" containerID="f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba" Oct 31 20:28:01.993497 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:28:01.993463 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba\": container with ID starting with f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba not found: ID does not exist" containerID="f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba" Oct 31 20:28:01.993609 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:01.993507 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba"} err="failed to get container status \"f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba\": rpc error: code = NotFound desc = could not find container \"f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba\": container with ID starting with f61d040b721095c3df1eab5e2a6562e85dc1a725fd44a05c19d75db3af8aaeba not found: ID does not exist" Oct 31 20:28:02.134592 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:02.134506 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv"] Oct 31 20:28:02.138016 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:02.137990 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-d28b6b/simulator-single-main-d28b6b-kserve-64b9bc4dc8-vg5lv"] Oct 31 20:28:02.485193 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:02.485164 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f0117a9-5192-4dfd-b305-b280a65255c6" path="/var/lib/kubelet/pods/9f0117a9-5192-4dfd-b305-b280a65255c6/volumes" Oct 31 20:28:03.456132 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:03.456097 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" podUID="b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:28:03.456132 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:03.456113 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" podUID="27f213fa-a773-4d24-a5f9-b9a66b76b449" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:28:13.455947 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:13.455905 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" podUID="b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:28:13.456342 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:13.455924 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" podUID="27f213fa-a773-4d24-a5f9-b9a66b76b449" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:28:16.968396 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:16.968361 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh"] Oct 31 20:28:16.968807 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:16.968760 6107 cpu_manager.go:414] "containerMap: removing container" podUID="9f0117a9-5192-4dfd-b305-b280a65255c6" containerName="main" Oct 31 20:28:16.968807 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:16.968770 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0117a9-5192-4dfd-b305-b280a65255c6" containerName="main" Oct 31 20:28:16.968911 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:16.968843 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0117a9-5192-4dfd-b305-b280a65255c6" containerName="main" Oct 31 20:28:16.971964 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:16.971945 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.064578 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.064541 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4fc33141-kserve-self-signed-certs\"" Oct 31 20:28:17.068683 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.068661 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh"] Oct 31 20:28:17.100360 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.100324 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-tmp-dir\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.100535 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.100455 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-model-cache\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.100604 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.100551 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-dshm\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.100681 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.100659 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zs2p\" (UniqueName: \"kubernetes.io/projected/1a14c264-0fa4-4789-883b-b994ae848be2-kube-api-access-9zs2p\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.100745 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.100729 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a14c264-0fa4-4789-883b-b994ae848be2-tls-certs\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.100802 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.100771 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-home\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.201339 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.201306 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-dshm\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.201534 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.201347 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zs2p\" (UniqueName: \"kubernetes.io/projected/1a14c264-0fa4-4789-883b-b994ae848be2-kube-api-access-9zs2p\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.201534 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.201367 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a14c264-0fa4-4789-883b-b994ae848be2-tls-certs\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.201619 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.201552 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-home\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.201664 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.201624 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-tmp-dir\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.201717 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.201697 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-model-cache\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.201973 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.201942 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-home\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.202102 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.201989 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-tmp-dir\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.202102 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.202029 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-model-cache\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.203670 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.203648 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-dshm\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.203833 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.203816 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a14c264-0fa4-4789-883b-b994ae848be2-tls-certs\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.261869 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.261840 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zs2p\" (UniqueName: \"kubernetes.io/projected/1a14c264-0fa4-4789-883b-b994ae848be2-kube-api-access-9zs2p\") pod \"e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.283550 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.283519 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:17.650578 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:17.650555 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh"] Oct 31 20:28:17.653692 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:28:17.653660 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a14c264_0fa4_4789_883b_b994ae848be2.slice/crio-5fb8b94dc9cf8c6e6f10f892cd608be2dc9324fe9c44e2370b5d84c53a1ac3b0 WatchSource:0}: Error finding container 5fb8b94dc9cf8c6e6f10f892cd608be2dc9324fe9c44e2370b5d84c53a1ac3b0: Status 404 returned error can't find the container with id 5fb8b94dc9cf8c6e6f10f892cd608be2dc9324fe9c44e2370b5d84c53a1ac3b0 Oct 31 20:28:18.061990 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.061946 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" event={"ID":"1a14c264-0fa4-4789-883b-b994ae848be2","Type":"ContainerStarted","Data":"375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551"} Oct 31 20:28:18.062495 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.062007 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" event={"ID":"1a14c264-0fa4-4789-883b-b994ae848be2","Type":"ContainerStarted","Data":"5fb8b94dc9cf8c6e6f10f892cd608be2dc9324fe9c44e2370b5d84c53a1ac3b0"} Oct 31 20:28:18.175461 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.175402 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" podStartSLOduration=2.175386494 podStartE2EDuration="2.175386494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:28:18.173962692 +0000 UTC m=+4269.807356601" watchObservedRunningTime="2026-10-31 20:28:18.175386494 +0000 UTC m=+4269.808780401" Oct 31 20:28:18.284212 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.284178 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:18.777732 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.777668 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" Oct 31 20:28:18.863642 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.863618 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" Oct 31 20:28:18.920614 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.920586 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f213fa-a773-4d24-a5f9-b9a66b76b449-kube-api-access-q2ncc\" (UniqueName: \"kubernetes.io/projected/27f213fa-a773-4d24-a5f9-b9a66b76b449-kube-api-access-q2ncc\") pod \"27f213fa-a773-4d24-a5f9-b9a66b76b449\" (UID: \"27f213fa-a773-4d24-a5f9-b9a66b76b449\") " Oct 31 20:28:18.920805 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.920721 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-plugins-config-volume\") pod \"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30\" (UID: \"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30\") " Oct 31 20:28:18.920805 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.920770 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-kube-api-access-24mlm\" (UniqueName: \"kubernetes.io/projected/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-kube-api-access-24mlm\") pod \"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30\" (UID: \"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30\") " Oct 31 20:28:18.920976 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.920807 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f213fa-a773-4d24-a5f9-b9a66b76b449-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f213fa-a773-4d24-a5f9-b9a66b76b449-plugins-config-volume\") pod \"27f213fa-a773-4d24-a5f9-b9a66b76b449\" (UID: \"27f213fa-a773-4d24-a5f9-b9a66b76b449\") " Oct 31 20:28:18.921109 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.921077 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-plugins-config-volume" pod "b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30" (UID: "b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:28:18.921229 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.921151 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f213fa-a773-4d24-a5f9-b9a66b76b449-plugins-config-volume" pod "27f213fa-a773-4d24-a5f9-b9a66b76b449" (UID: "27f213fa-a773-4d24-a5f9-b9a66b76b449"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:28:18.921281 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.921223 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:18.922685 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.922667 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f213fa-a773-4d24-a5f9-b9a66b76b449-kube-api-access-q2ncc" pod "27f213fa-a773-4d24-a5f9-b9a66b76b449" (UID: "27f213fa-a773-4d24-a5f9-b9a66b76b449"). InnerVolumeSpecName "kube-api-access-q2ncc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:28:18.922750 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:18.922710 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-kube-api-access-24mlm" pod "b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30" (UID: "b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30"). InnerVolumeSpecName "kube-api-access-24mlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:28:19.022272 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.022231 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24mlm\" (UniqueName: \"kubernetes.io/projected/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30-kube-api-access-24mlm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:19.022272 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.022258 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f213fa-a773-4d24-a5f9-b9a66b76b449-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:19.022272 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.022267 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2ncc\" (UniqueName: \"kubernetes.io/projected/27f213fa-a773-4d24-a5f9-b9a66b76b449-kube-api-access-q2ncc\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:19.068326 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.068245 6107 generic.go:356] "Generic (PLEG): container finished" podID="b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30" containerID="68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2" exitCode=0 Oct 31 20:28:19.068326 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.068294 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" Oct 31 20:28:19.068827 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.068334 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" event={"ID":"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30","Type":"ContainerDied","Data":"68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2"} Oct 31 20:28:19.068827 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.068387 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b" event={"ID":"b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30","Type":"ContainerDied","Data":"cbca53c3ba90b268ca9561397da31c5798c711ab9eec74b6af082057e41a1c9e"} Oct 31 20:28:19.068827 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.068409 6107 scope.go:122] "RemoveContainer" containerID="68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2" Oct 31 20:28:19.069871 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.069852 6107 generic.go:356] "Generic (PLEG): container finished" podID="27f213fa-a773-4d24-a5f9-b9a66b76b449" containerID="96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f" exitCode=0 Oct 31 20:28:19.070011 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.069914 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" Oct 31 20:28:19.070011 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.069943 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" event={"ID":"27f213fa-a773-4d24-a5f9-b9a66b76b449","Type":"ContainerDied","Data":"96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f"} Oct 31 20:28:19.070011 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.069994 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn" event={"ID":"27f213fa-a773-4d24-a5f9-b9a66b76b449","Type":"ContainerDied","Data":"26b09f9c783b0db3a751433bfaa470efd6022819afd9e9b4bf08368b5fa3088f"} Oct 31 20:28:19.077208 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.077189 6107 scope.go:122] "RemoveContainer" containerID="68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2" Oct 31 20:28:19.077493 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:28:19.077473 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2\": container with ID starting with 68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2 not found: ID does not exist" containerID="68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2" Oct 31 20:28:19.077579 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.077500 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2"} err="failed to get container status \"68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2\": rpc error: code = NotFound desc = could not find container \"68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2\": container with ID starting with 68e78d757eb4082b639490ce09c069989406684cf91ea25e07d16c1537d763b2 not found: ID does not exist" Oct 31 20:28:19.077579 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.077524 6107 scope.go:122] "RemoveContainer" containerID="96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f" Oct 31 20:28:19.085555 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.085529 6107 scope.go:122] "RemoveContainer" containerID="96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f" Oct 31 20:28:19.085808 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:28:19.085789 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f\": container with ID starting with 96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f not found: ID does not exist" containerID="96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f" Oct 31 20:28:19.085855 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.085816 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f"} err="failed to get container status \"96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f\": rpc error: code = NotFound desc = could not find container \"96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f\": container with ID starting with 96e0a4f05f5c5b5f0938287b30c747febf2439b924c51c1cb7cd15cdf98a706f not found: ID does not exist" Oct 31 20:28:19.219331 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.219305 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn"] Oct 31 20:28:19.223374 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.223353 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d28b6b-57749n2tsn"] Oct 31 20:28:19.323099 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.323011 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b"] Oct 31 20:28:19.327125 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:19.327099 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d28b6b-7d8df5c98lxp4b"] Oct 31 20:28:20.402697 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:28:20.402662 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:28:20.485557 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:20.485520 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f213fa-a773-4d24-a5f9-b9a66b76b449" path="/var/lib/kubelet/pods/27f213fa-a773-4d24-a5f9-b9a66b76b449/volumes" Oct 31 20:28:20.485957 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:20.485941 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30" path="/var/lib/kubelet/pods/b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30/volumes" Oct 31 20:28:24.205906 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:24.204988 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-jkrrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:28:24.205906 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:24.205054 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-jkrrj" podUID="dca75e6d-22eb-42ed-9d6c-4edb0d5678ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:28:27.284180 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:27.284141 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:27.296826 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:27.296787 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:28.127328 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:28.127291 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:28:50.401939 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:28:50.401901 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:28:59.691035 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.690994 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86"] Oct 31 20:28:59.691489 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.691409 6107 cpu_manager.go:414] "containerMap: removing container" podUID="b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30" containerName="payload-processing" Oct 31 20:28:59.691489 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.691422 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30" containerName="payload-processing" Oct 31 20:28:59.691489 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.691435 6107 cpu_manager.go:414] "containerMap: removing container" podUID="27f213fa-a773-4d24-a5f9-b9a66b76b449" containerName="payload-pre-processing" Oct 31 20:28:59.691489 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.691440 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f213fa-a773-4d24-a5f9-b9a66b76b449" containerName="payload-pre-processing" Oct 31 20:28:59.691640 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.691517 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b6a8f2-b3ea-4b35-83ca-25fa491e1b30" containerName="payload-processing" Oct 31 20:28:59.691640 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.691524 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f213fa-a773-4d24-a5f9-b9a66b76b449" containerName="payload-pre-processing" Oct 31 20:28:59.694683 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.694661 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.743237 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.743211 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4fc33141-kserve-self-signed-certs\"" Oct 31 20:28:59.791017 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.790989 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86"] Oct 31 20:28:59.843694 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.843662 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92sv\" (UniqueName: \"kubernetes.io/projected/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-kube-api-access-c92sv\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.843871 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.843738 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tls-certs\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.843871 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.843857 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-home\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.844037 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.843910 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-dshm\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.844037 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.843987 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-model-cache\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.844037 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.844028 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tmp-dir\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.944872 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.944794 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tls-certs\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.944872 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.944838 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-home\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.944872 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.944855 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-dshm\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.945165 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.944905 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-model-cache\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.945165 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.944933 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tmp-dir\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.945165 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.945004 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92sv\" (UniqueName: \"kubernetes.io/projected/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-kube-api-access-c92sv\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.945341 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.945299 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-home\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.945400 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.945380 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tmp-dir\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.945400 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.945385 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-model-cache\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.947187 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.947158 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-dshm\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.947477 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.947460 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tls-certs\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:28:59.997536 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:28:59.997502 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92sv\" (UniqueName: \"kubernetes.io/projected/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-kube-api-access-c92sv\") pod \"e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:29:00.006376 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:00.006354 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:29:00.369592 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:00.369565 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86"] Oct 31 20:29:00.371498 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:29:00.371465 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4ea2ffa_0f9d_4fc1_ac50_489cd1c4d3c8.slice/crio-7f408b83e7e23a85cf09ccfb1aa687aa8c1dd8ca082a09f5c70c2c37329a0178 WatchSource:0}: Error finding container 7f408b83e7e23a85cf09ccfb1aa687aa8c1dd8ca082a09f5c70c2c37329a0178: Status 404 returned error can't find the container with id 7f408b83e7e23a85cf09ccfb1aa687aa8c1dd8ca082a09f5c70c2c37329a0178 Oct 31 20:29:01.282899 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:01.282855 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" event={"ID":"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8","Type":"ContainerStarted","Data":"48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806"} Oct 31 20:29:01.283292 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:01.282913 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" event={"ID":"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8","Type":"ContainerStarted","Data":"7f408b83e7e23a85cf09ccfb1aa687aa8c1dd8ca082a09f5c70c2c37329a0178"} Oct 31 20:29:01.389706 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:01.389654 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" podStartSLOduration=2.389642249 podStartE2EDuration="2.389642249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:29:01.389379903 +0000 UTC m=+4313.022773810" watchObservedRunningTime="2026-10-31 20:29:01.389642249 +0000 UTC m=+4313.023036157" Oct 31 20:29:02.007012 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:02.006967 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:29:10.006543 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:10.006513 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:29:10.018892 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:10.018853 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:29:10.377269 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:10.377187 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:29:20.403083 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:29:20.402994 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:29:37.691312 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:37.691015 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zjsq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:29:37.691312 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:37.691068 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" podUID="a5360256-7648-404e-ae6c-e5556cca1ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:29:46.674922 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:46.674875 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh"] Oct 31 20:29:50.403251 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:29:50.403213 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:29:52.680023 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:52.679960 6107 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zjsq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:29:52.680643 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:52.680613 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zjsq8" podUID="a5360256-7648-404e-ae6c-e5556cca1ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:29:53.238903 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:29:53.238350 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86"] Oct 31 20:30:01.694023 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.693985 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" podUID="1a14c264-0fa4-4789-883b-b994ae848be2" containerName="main" containerID="cri-o://375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551" gracePeriod=45 Oct 31 20:30:01.939539 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.939513 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:30:01.998670 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.998644 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-tmp-dir\") pod \"1a14c264-0fa4-4789-883b-b994ae848be2\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " Oct 31 20:30:01.998876 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.998686 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a14c264-0fa4-4789-883b-b994ae848be2-kube-api-access-9zs2p\" (UniqueName: \"kubernetes.io/projected/1a14c264-0fa4-4789-883b-b994ae848be2-kube-api-access-9zs2p\") pod \"1a14c264-0fa4-4789-883b-b994ae848be2\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " Oct 31 20:30:01.998876 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.998732 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-model-cache\") pod \"1a14c264-0fa4-4789-883b-b994ae848be2\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " Oct 31 20:30:01.998876 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.998764 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a14c264-0fa4-4789-883b-b994ae848be2-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a14c264-0fa4-4789-883b-b994ae848be2-tls-certs\") pod \"1a14c264-0fa4-4789-883b-b994ae848be2\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " Oct 31 20:30:01.998876 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.998841 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-dshm\") pod \"1a14c264-0fa4-4789-883b-b994ae848be2\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " Oct 31 20:30:01.998876 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.998875 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-home\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-home\") pod \"1a14c264-0fa4-4789-883b-b994ae848be2\" (UID: \"1a14c264-0fa4-4789-883b-b994ae848be2\") " Oct 31 20:30:01.999199 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.998966 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-tmp-dir" pod "1a14c264-0fa4-4789-883b-b994ae848be2" (UID: "1a14c264-0fa4-4789-883b-b994ae848be2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:30:01.999199 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.999045 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-model-cache" pod "1a14c264-0fa4-4789-883b-b994ae848be2" (UID: "1a14c264-0fa4-4789-883b-b994ae848be2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:30:01.999289 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.999209 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-home" pod "1a14c264-0fa4-4789-883b-b994ae848be2" (UID: "1a14c264-0fa4-4789-883b-b994ae848be2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:30:01.999289 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.999217 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:01.999289 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:01.999280 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:02.001114 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.001090 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-dshm" pod "1a14c264-0fa4-4789-883b-b994ae848be2" (UID: "1a14c264-0fa4-4789-883b-b994ae848be2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:30:02.001220 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.001118 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a14c264-0fa4-4789-883b-b994ae848be2-tls-certs" pod "1a14c264-0fa4-4789-883b-b994ae848be2" (UID: "1a14c264-0fa4-4789-883b-b994ae848be2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:30:02.001220 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.001123 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a14c264-0fa4-4789-883b-b994ae848be2-kube-api-access-9zs2p" pod "1a14c264-0fa4-4789-883b-b994ae848be2" (UID: "1a14c264-0fa4-4789-883b-b994ae848be2"). InnerVolumeSpecName "kube-api-access-9zs2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:30:02.100104 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.100071 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:02.100104 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.100099 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a14c264-0fa4-4789-883b-b994ae848be2-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:02.100104 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.100108 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zs2p\" (UniqueName: \"kubernetes.io/projected/1a14c264-0fa4-4789-883b-b994ae848be2-kube-api-access-9zs2p\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:02.100333 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.100117 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a14c264-0fa4-4789-883b-b994ae848be2-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:02.674538 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.674503 6107 generic.go:356] "Generic (PLEG): container finished" podID="1a14c264-0fa4-4789-883b-b994ae848be2" containerID="375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551" exitCode=0 Oct 31 20:30:02.674724 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.674583 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" Oct 31 20:30:02.674724 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.674588 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" event={"ID":"1a14c264-0fa4-4789-883b-b994ae848be2","Type":"ContainerDied","Data":"375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551"} Oct 31 20:30:02.674724 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.674644 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh" event={"ID":"1a14c264-0fa4-4789-883b-b994ae848be2","Type":"ContainerDied","Data":"5fb8b94dc9cf8c6e6f10f892cd608be2dc9324fe9c44e2370b5d84c53a1ac3b0"} Oct 31 20:30:02.674724 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.674669 6107 scope.go:122] "RemoveContainer" containerID="375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551" Oct 31 20:30:02.683804 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.683785 6107 scope.go:122] "RemoveContainer" containerID="375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551" Oct 31 20:30:02.684089 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:30:02.684067 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551\": container with ID starting with 375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551 not found: ID does not exist" containerID="375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551" Oct 31 20:30:02.684195 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.684098 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551"} err="failed to get container status \"375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551\": rpc error: code = NotFound desc = could not find container \"375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551\": container with ID starting with 375280a452a0daac4d5f0ab3cef405c4822f95bcfafc79762527287bae6a4551 not found: ID does not exist" Oct 31 20:30:02.827700 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.827618 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh"] Oct 31 20:30:02.874183 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:02.874150 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4fc33141-kserve-5fff66d669-kk8vh"] Oct 31 20:30:04.484996 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:04.484963 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a14c264-0fa4-4789-883b-b994ae848be2" path="/var/lib/kubelet/pods/1a14c264-0fa4-4789-883b-b994ae848be2/volumes" Oct 31 20:30:08.288292 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.288250 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" podUID="b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" containerName="main" containerID="cri-o://48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806" gracePeriod=45 Oct 31 20:30:08.367767 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.367728 6107 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" podUID="b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": dial tcp 10.133.0.141:8000: connect: connection refused" Oct 31 20:30:08.537750 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.537726 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:30:08.666324 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.666246 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-kube-api-access-c92sv\" (UniqueName: \"kubernetes.io/projected/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-kube-api-access-c92sv\") pod \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " Oct 31 20:30:08.666324 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.666299 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tls-certs\") pod \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " Oct 31 20:30:08.666568 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.666344 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-dshm\") pod \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " Oct 31 20:30:08.666568 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.666366 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tmp-dir\") pod \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " Oct 31 20:30:08.666568 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.666418 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-model-cache\") pod \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " Oct 31 20:30:08.666568 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.666454 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-home\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-home\") pod \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\" (UID: \"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8\") " Oct 31 20:30:08.666816 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.666663 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tmp-dir" pod "b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" (UID: "b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:30:08.666816 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.666673 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-model-cache" pod "b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" (UID: "b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:30:08.666972 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.666917 6107 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tmp-dir\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:08.666972 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.666925 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-home" pod "b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" (UID: "b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:30:08.666972 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.666939 6107 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-model-cache\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:08.668414 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.668384 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-kube-api-access-c92sv" pod "b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" (UID: "b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8"). InnerVolumeSpecName "kube-api-access-c92sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:30:08.668518 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.668461 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tls-certs" pod "b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" (UID: "b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:30:08.668559 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.668544 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-dshm" pod "b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" (UID: "b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:30:08.706617 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.706589 6107 generic.go:356] "Generic (PLEG): container finished" podID="b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" containerID="48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806" exitCode=0 Oct 31 20:30:08.706745 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.706649 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" Oct 31 20:30:08.706745 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.706667 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" event={"ID":"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8","Type":"ContainerDied","Data":"48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806"} Oct 31 20:30:08.706745 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.706713 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86" event={"ID":"b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8","Type":"ContainerDied","Data":"7f408b83e7e23a85cf09ccfb1aa687aa8c1dd8ca082a09f5c70c2c37329a0178"} Oct 31 20:30:08.706745 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.706730 6107 scope.go:122] "RemoveContainer" containerID="48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806" Oct 31 20:30:08.715446 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.715431 6107 scope.go:122] "RemoveContainer" containerID="48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806" Oct 31 20:30:08.715724 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:30:08.715694 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806\": container with ID starting with 48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806 not found: ID does not exist" containerID="48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806" Oct 31 20:30:08.715767 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.715731 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806"} err="failed to get container status \"48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806\": rpc error: code = NotFound desc = could not find container \"48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806\": container with ID starting with 48092d6c99f918a2a0c92ff41318e6dd4425babbf296c82c5bf0845c7160f806 not found: ID does not exist" Oct 31 20:30:08.767669 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.767641 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c92sv\" (UniqueName: \"kubernetes.io/projected/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-kube-api-access-c92sv\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:08.767669 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.767666 6107 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-tls-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:08.767669 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.767676 6107 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-dshm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:08.767911 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.767686 6107 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8-home\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:08.860028 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.859996 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86"] Oct 31 20:30:08.863925 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:08.863895 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4fc33141-kserve-5f9bb88b85-tqj86"] Oct 31 20:30:10.485667 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:10.485636 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" path="/var/lib/kubelet/pods/b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8/volumes" Oct 31 20:30:20.403783 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:30:20.403748 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:30:36.579609 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.579572 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh"] Oct 31 20:30:36.580101 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.579989 6107 cpu_manager.go:414] "containerMap: removing container" podUID="1a14c264-0fa4-4789-883b-b994ae848be2" containerName="main" Oct 31 20:30:36.580101 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.580001 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a14c264-0fa4-4789-883b-b994ae848be2" containerName="main" Oct 31 20:30:36.580101 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.580019 6107 cpu_manager.go:414] "containerMap: removing container" podUID="b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" containerName="main" Oct 31 20:30:36.580101 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.580024 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" containerName="main" Oct 31 20:30:36.580101 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.580077 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a14c264-0fa4-4789-883b-b994ae848be2" containerName="main" Oct 31 20:30:36.580101 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.580088 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ea2ffa-0f9d-4fc1-ac50-489cd1c4d3c8" containerName="main" Oct 31 20:30:36.583215 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.583196 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.673340 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.673305 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfmj8\" (UniqueName: \"kubernetes.io/projected/c83c38f0-bc64-4713-9299-f8251aa4333a-kube-api-access-mfmj8\") pod \"maas-api-74f5c976df-bl8kh\" (UID: \"c83c38f0-bc64-4713-9299-f8251aa4333a\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.673340 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.673351 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83c38f0-bc64-4713-9299-f8251aa4333a-openshift-service-ca\") pod \"maas-api-74f5c976df-bl8kh\" (UID: \"c83c38f0-bc64-4713-9299-f8251aa4333a\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.673579 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.673369 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83c38f0-bc64-4713-9299-f8251aa4333a-metrics-certs\") pod \"maas-api-74f5c976df-bl8kh\" (UID: \"c83c38f0-bc64-4713-9299-f8251aa4333a\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.673579 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.673483 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c83c38f0-bc64-4713-9299-f8251aa4333a-maas-api-tls\") pod \"maas-api-74f5c976df-bl8kh\" (UID: \"c83c38f0-bc64-4713-9299-f8251aa4333a\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.680299 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.680270 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh"] Oct 31 20:30:36.774848 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.774812 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfmj8\" (UniqueName: \"kubernetes.io/projected/c83c38f0-bc64-4713-9299-f8251aa4333a-kube-api-access-mfmj8\") pod \"maas-api-74f5c976df-bl8kh\" (UID: \"c83c38f0-bc64-4713-9299-f8251aa4333a\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.774848 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.774853 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83c38f0-bc64-4713-9299-f8251aa4333a-openshift-service-ca\") pod \"maas-api-74f5c976df-bl8kh\" (UID: \"c83c38f0-bc64-4713-9299-f8251aa4333a\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.775100 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.774871 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83c38f0-bc64-4713-9299-f8251aa4333a-metrics-certs\") pod \"maas-api-74f5c976df-bl8kh\" (UID: \"c83c38f0-bc64-4713-9299-f8251aa4333a\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.775100 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.774916 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c83c38f0-bc64-4713-9299-f8251aa4333a-maas-api-tls\") pod \"maas-api-74f5c976df-bl8kh\" (UID: \"c83c38f0-bc64-4713-9299-f8251aa4333a\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.775585 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.775561 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83c38f0-bc64-4713-9299-f8251aa4333a-openshift-service-ca\") pod \"maas-api-74f5c976df-bl8kh\" (UID: \"c83c38f0-bc64-4713-9299-f8251aa4333a\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.777343 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.777320 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83c38f0-bc64-4713-9299-f8251aa4333a-metrics-certs\") pod \"maas-api-74f5c976df-bl8kh\" (UID: \"c83c38f0-bc64-4713-9299-f8251aa4333a\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.777516 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.777487 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c83c38f0-bc64-4713-9299-f8251aa4333a-maas-api-tls\") pod \"maas-api-74f5c976df-bl8kh\" (UID: \"c83c38f0-bc64-4713-9299-f8251aa4333a\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.827546 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.827505 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfmj8\" (UniqueName: \"kubernetes.io/projected/c83c38f0-bc64-4713-9299-f8251aa4333a-kube-api-access-mfmj8\") pod \"maas-api-74f5c976df-bl8kh\" (UID: \"c83c38f0-bc64-4713-9299-f8251aa4333a\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:36.895808 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:36.895731 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:37.252729 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:37.252704 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh"] Oct 31 20:30:37.255524 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:30:37.255491 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83c38f0_bc64_4713_9299_f8251aa4333a.slice/crio-7653059c8409780b33c289a71dc32f184495c0ff38ea5ba0ac1ca079a1fcfca8 WatchSource:0}: Error finding container 7653059c8409780b33c289a71dc32f184495c0ff38ea5ba0ac1ca079a1fcfca8: Status 404 returned error can't find the container with id 7653059c8409780b33c289a71dc32f184495c0ff38ea5ba0ac1ca079a1fcfca8 Oct 31 20:30:37.256926 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:37.256908 6107 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 20:30:37.876261 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:37.876224 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" event={"ID":"c83c38f0-bc64-4713-9299-f8251aa4333a","Type":"ContainerStarted","Data":"7653059c8409780b33c289a71dc32f184495c0ff38ea5ba0ac1ca079a1fcfca8"} Oct 31 20:30:38.890515 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:38.890479 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" event={"ID":"c83c38f0-bc64-4713-9299-f8251aa4333a","Type":"ContainerStarted","Data":"5d128404f30ee1dd5b82081fa5afb81bee378c2e703b0e64cc1ffc6dbad22735"} Oct 31 20:30:38.890967 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:38.890656 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:38.995216 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:38.995164 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" podStartSLOduration=2.102259975 podStartE2EDuration="2.995151808s" totalImagesPullingTime="892.891833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:30:38.993926532 +0000 UTC m=+4410.627320437" watchObservedRunningTime="2026-10-31 20:30:38.995151808 +0000 UTC m=+4410.628545719" Oct 31 20:30:44.901553 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:44.901521 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74f5c976df-bl8kh" Oct 31 20:30:50.406773 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:30:50.406683 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:30:51.016251 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.016209 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9"] Oct 31 20:30:51.019776 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.019750 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" Oct 31 20:30:51.072511 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.072472 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-47884b3f\"" Oct 31 20:30:51.117937 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.117900 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-47884b3f-dockercfg-975sl\"" Oct 31 20:30:51.118511 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.118459 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95xdm\" (UniqueName: \"kubernetes.io/projected/56adc916-d73a-4604-b371-1d5ea37b9f50-kube-api-access-95xdm\") pod \"payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9\" (UID: \"56adc916-d73a-4604-b371-1d5ea37b9f50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" Oct 31 20:30:51.118626 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.118514 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56adc916-d73a-4604-b371-1d5ea37b9f50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9\" (UID: \"56adc916-d73a-4604-b371-1d5ea37b9f50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" Oct 31 20:30:51.120168 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.120140 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9"] Oct 31 20:30:51.191194 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.191153 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb"] Oct 31 20:30:51.194855 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.194830 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" Oct 31 20:30:51.220210 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.220166 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlz8p\" (UniqueName: \"kubernetes.io/projected/5a4c7305-7cec-4351-aaf0-8c0cea11370d-kube-api-access-rlz8p\") pod \"payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb\" (UID: \"5a4c7305-7cec-4351-aaf0-8c0cea11370d\") " pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" Oct 31 20:30:51.220410 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.220340 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95xdm\" (UniqueName: \"kubernetes.io/projected/56adc916-d73a-4604-b371-1d5ea37b9f50-kube-api-access-95xdm\") pod \"payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9\" (UID: \"56adc916-d73a-4604-b371-1d5ea37b9f50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" Oct 31 20:30:51.220410 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.220384 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56adc916-d73a-4604-b371-1d5ea37b9f50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9\" (UID: \"56adc916-d73a-4604-b371-1d5ea37b9f50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" Oct 31 20:30:51.220566 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.220478 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4c7305-7cec-4351-aaf0-8c0cea11370d-plugins-config-volume\") pod \"payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb\" (UID: \"5a4c7305-7cec-4351-aaf0-8c0cea11370d\") " pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" Oct 31 20:30:51.221350 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.221327 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56adc916-d73a-4604-b371-1d5ea37b9f50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9\" (UID: \"56adc916-d73a-4604-b371-1d5ea37b9f50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" Oct 31 20:30:51.253722 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.253686 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb"] Oct 31 20:30:51.292483 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.292413 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95xdm\" (UniqueName: \"kubernetes.io/projected/56adc916-d73a-4604-b371-1d5ea37b9f50-kube-api-access-95xdm\") pod \"payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9\" (UID: \"56adc916-d73a-4604-b371-1d5ea37b9f50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" Oct 31 20:30:51.321181 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.321142 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlz8p\" (UniqueName: \"kubernetes.io/projected/5a4c7305-7cec-4351-aaf0-8c0cea11370d-kube-api-access-rlz8p\") pod \"payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb\" (UID: \"5a4c7305-7cec-4351-aaf0-8c0cea11370d\") " pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" Oct 31 20:30:51.321366 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.321256 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4c7305-7cec-4351-aaf0-8c0cea11370d-plugins-config-volume\") pod \"payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb\" (UID: \"5a4c7305-7cec-4351-aaf0-8c0cea11370d\") " pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" Oct 31 20:30:51.321933 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.321915 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4c7305-7cec-4351-aaf0-8c0cea11370d-plugins-config-volume\") pod \"payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb\" (UID: \"5a4c7305-7cec-4351-aaf0-8c0cea11370d\") " pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" Oct 31 20:30:51.334056 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.334023 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" Oct 31 20:30:51.377053 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.376981 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlz8p\" (UniqueName: \"kubernetes.io/projected/5a4c7305-7cec-4351-aaf0-8c0cea11370d-kube-api-access-rlz8p\") pod \"payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb\" (UID: \"5a4c7305-7cec-4351-aaf0-8c0cea11370d\") " pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" Oct 31 20:30:51.507743 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.507698 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" Oct 31 20:30:51.706532 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.706502 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9"] Oct 31 20:30:51.708966 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:30:51.708938 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56adc916_d73a_4604_b371_1d5ea37b9f50.slice/crio-2995e3b0ff8fd9bb09591c1977380134c811dce47c6fb8db6ac21ec9ff0f8073 WatchSource:0}: Error finding container 2995e3b0ff8fd9bb09591c1977380134c811dce47c6fb8db6ac21ec9ff0f8073: Status 404 returned error can't find the container with id 2995e3b0ff8fd9bb09591c1977380134c811dce47c6fb8db6ac21ec9ff0f8073 Oct 31 20:30:51.877673 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.877649 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb"] Oct 31 20:30:51.880204 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:30:51.880166 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4c7305_7cec_4351_aaf0_8c0cea11370d.slice/crio-7f94df6e54a10af39f57040f196cb2e8997f0aceb9d6c796ea9c379929cb843b WatchSource:0}: Error finding container 7f94df6e54a10af39f57040f196cb2e8997f0aceb9d6c796ea9c379929cb843b: Status 404 returned error can't find the container with id 7f94df6e54a10af39f57040f196cb2e8997f0aceb9d6c796ea9c379929cb843b Oct 31 20:30:51.955919 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.955854 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" event={"ID":"5a4c7305-7cec-4351-aaf0-8c0cea11370d","Type":"ContainerStarted","Data":"a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e"} Oct 31 20:30:51.956386 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.955937 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" event={"ID":"5a4c7305-7cec-4351-aaf0-8c0cea11370d","Type":"ContainerStarted","Data":"7f94df6e54a10af39f57040f196cb2e8997f0aceb9d6c796ea9c379929cb843b"} Oct 31 20:30:51.956386 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.956058 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" Oct 31 20:30:51.957459 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.957432 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" event={"ID":"56adc916-d73a-4604-b371-1d5ea37b9f50","Type":"ContainerStarted","Data":"e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2"} Oct 31 20:30:51.957590 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.957473 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" event={"ID":"56adc916-d73a-4604-b371-1d5ea37b9f50","Type":"ContainerStarted","Data":"2995e3b0ff8fd9bb09591c1977380134c811dce47c6fb8db6ac21ec9ff0f8073"} Oct 31 20:30:51.957590 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:51.957495 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" Oct 31 20:30:52.068531 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:52.068437 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" podStartSLOduration=1.068418001 podStartE2EDuration="1.068418001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:30:52.06519329 +0000 UTC m=+4423.698587198" watchObservedRunningTime="2026-10-31 20:30:52.068418001 +0000 UTC m=+4423.701811914" Oct 31 20:30:52.172333 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:52.171747 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" podStartSLOduration=2.171727122 podStartE2EDuration="2.171727122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:30:52.171514663 +0000 UTC m=+4423.804908573" watchObservedRunningTime="2026-10-31 20:30:52.171727122 +0000 UTC m=+4423.805121030" Oct 31 20:30:53.301204 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:53.301163 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77"] Oct 31 20:30:53.301722 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:53.301435 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" podUID="373c6bd5-cd00-43f7-b515-7aa8606a2bc6" containerName="maas-api" containerID="cri-o://b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485" gracePeriod=30 Oct 31 20:30:54.565834 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.565796 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 20:30:54.657225 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.657130 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-metrics-certs\") pod \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " Oct 31 20:30:54.657393 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.657310 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-maas-api-tls\") pod \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " Oct 31 20:30:54.657393 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.657337 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-openshift-service-ca\") pod \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " Oct 31 20:30:54.657393 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.657383 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-kube-api-access-72fv2\" (UniqueName: \"kubernetes.io/projected/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-kube-api-access-72fv2\") pod \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\" (UID: \"373c6bd5-cd00-43f7-b515-7aa8606a2bc6\") " Oct 31 20:30:54.657784 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.657743 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-openshift-service-ca" pod "373c6bd5-cd00-43f7-b515-7aa8606a2bc6" (UID: "373c6bd5-cd00-43f7-b515-7aa8606a2bc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:30:54.659429 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.659405 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-maas-api-tls" pod "373c6bd5-cd00-43f7-b515-7aa8606a2bc6" (UID: "373c6bd5-cd00-43f7-b515-7aa8606a2bc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:30:54.659808 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.659782 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-metrics-certs" pod "373c6bd5-cd00-43f7-b515-7aa8606a2bc6" (UID: "373c6bd5-cd00-43f7-b515-7aa8606a2bc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:30:54.659915 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.659800 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-kube-api-access-72fv2" pod "373c6bd5-cd00-43f7-b515-7aa8606a2bc6" (UID: "373c6bd5-cd00-43f7-b515-7aa8606a2bc6"). InnerVolumeSpecName "kube-api-access-72fv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:30:54.759079 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.759041 6107 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-maas-api-tls\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:54.759079 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.759080 6107 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-openshift-service-ca\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:54.759079 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.759095 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72fv2\" (UniqueName: \"kubernetes.io/projected/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-kube-api-access-72fv2\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:54.759371 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.759107 6107 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373c6bd5-cd00-43f7-b515-7aa8606a2bc6-metrics-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:54.974414 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.974380 6107 generic.go:356] "Generic (PLEG): container finished" podID="373c6bd5-cd00-43f7-b515-7aa8606a2bc6" containerID="b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485" exitCode=0 Oct 31 20:30:54.974602 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.974471 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" Oct 31 20:30:54.974602 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.974469 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" event={"ID":"373c6bd5-cd00-43f7-b515-7aa8606a2bc6","Type":"ContainerDied","Data":"b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485"} Oct 31 20:30:54.974687 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.974619 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77" event={"ID":"373c6bd5-cd00-43f7-b515-7aa8606a2bc6","Type":"ContainerDied","Data":"767055c1235c013a6af4c7c1930c9fd7352331d30bba288821b20e67026d40ee"} Oct 31 20:30:54.974687 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.974640 6107 scope.go:122] "RemoveContainer" containerID="b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485" Oct 31 20:30:54.984467 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.984413 6107 scope.go:122] "RemoveContainer" containerID="b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485" Oct 31 20:30:54.984920 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:30:54.984876 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485\": container with ID starting with b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485 not found: ID does not exist" containerID="b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485" Oct 31 20:30:54.985031 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:54.984931 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485"} err="failed to get container status \"b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485\": rpc error: code = NotFound desc = could not find container \"b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485\": container with ID starting with b1f08cb77f6adc6e3cd4d0de257fbfbd08418fe8c28ea5803963231aaa435485 not found: ID does not exist" Oct 31 20:30:55.125472 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:55.125437 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77"] Oct 31 20:30:55.171508 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:55.171469 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d5f466c97-kqk77"] Oct 31 20:30:56.487124 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:30:56.487088 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373c6bd5-cd00-43f7-b515-7aa8606a2bc6" path="/var/lib/kubelet/pods/373c6bd5-cd00-43f7-b515-7aa8606a2bc6/volumes" Oct 31 20:31:02.965551 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:02.965519 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" Oct 31 20:31:02.965999 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:02.965604 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" Oct 31 20:31:20.404734 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:31:20.404698 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:31:35.234011 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:35.233957 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb"] Oct 31 20:31:35.341479 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:35.341438 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9"] Oct 31 20:31:40.235366 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.235297 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" podUID="5a4c7305-7cec-4351-aaf0-8c0cea11370d" containerName="payload-processing" containerID="cri-o://a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e" gracePeriod=25 Oct 31 20:31:40.342228 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.342176 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" podUID="56adc916-d73a-4604-b371-1d5ea37b9f50" containerName="payload-pre-processing" containerID="cri-o://e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2" gracePeriod=25 Oct 31 20:31:40.496086 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.496023 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" Oct 31 20:31:40.604330 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.604298 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" Oct 31 20:31:40.618139 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.618109 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4c7305-7cec-4351-aaf0-8c0cea11370d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4c7305-7cec-4351-aaf0-8c0cea11370d-plugins-config-volume\") pod \"5a4c7305-7cec-4351-aaf0-8c0cea11370d\" (UID: \"5a4c7305-7cec-4351-aaf0-8c0cea11370d\") " Oct 31 20:31:40.618340 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.618149 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4c7305-7cec-4351-aaf0-8c0cea11370d-kube-api-access-rlz8p\" (UniqueName: \"kubernetes.io/projected/5a4c7305-7cec-4351-aaf0-8c0cea11370d-kube-api-access-rlz8p\") pod \"5a4c7305-7cec-4351-aaf0-8c0cea11370d\" (UID: \"5a4c7305-7cec-4351-aaf0-8c0cea11370d\") " Oct 31 20:31:40.618433 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.618409 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4c7305-7cec-4351-aaf0-8c0cea11370d-plugins-config-volume" pod "5a4c7305-7cec-4351-aaf0-8c0cea11370d" (UID: "5a4c7305-7cec-4351-aaf0-8c0cea11370d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:31:40.618649 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.618616 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4c7305-7cec-4351-aaf0-8c0cea11370d-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:40.620516 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.620494 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4c7305-7cec-4351-aaf0-8c0cea11370d-kube-api-access-rlz8p" pod "5a4c7305-7cec-4351-aaf0-8c0cea11370d" (UID: "5a4c7305-7cec-4351-aaf0-8c0cea11370d"). InnerVolumeSpecName "kube-api-access-rlz8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:31:40.719414 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.719376 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56adc916-d73a-4604-b371-1d5ea37b9f50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56adc916-d73a-4604-b371-1d5ea37b9f50-plugins-config-volume\") pod \"56adc916-d73a-4604-b371-1d5ea37b9f50\" (UID: \"56adc916-d73a-4604-b371-1d5ea37b9f50\") " Oct 31 20:31:40.719615 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.719429 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56adc916-d73a-4604-b371-1d5ea37b9f50-kube-api-access-95xdm\" (UniqueName: \"kubernetes.io/projected/56adc916-d73a-4604-b371-1d5ea37b9f50-kube-api-access-95xdm\") pod \"56adc916-d73a-4604-b371-1d5ea37b9f50\" (UID: \"56adc916-d73a-4604-b371-1d5ea37b9f50\") " Oct 31 20:31:40.719793 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.719765 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlz8p\" (UniqueName: \"kubernetes.io/projected/5a4c7305-7cec-4351-aaf0-8c0cea11370d-kube-api-access-rlz8p\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:40.719793 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.719777 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56adc916-d73a-4604-b371-1d5ea37b9f50-plugins-config-volume" pod "56adc916-d73a-4604-b371-1d5ea37b9f50" (UID: "56adc916-d73a-4604-b371-1d5ea37b9f50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:31:40.721485 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.721463 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56adc916-d73a-4604-b371-1d5ea37b9f50-kube-api-access-95xdm" pod "56adc916-d73a-4604-b371-1d5ea37b9f50" (UID: "56adc916-d73a-4604-b371-1d5ea37b9f50"). InnerVolumeSpecName "kube-api-access-95xdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:31:40.820977 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.820866 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56adc916-d73a-4604-b371-1d5ea37b9f50-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:40.820977 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:40.820924 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95xdm\" (UniqueName: \"kubernetes.io/projected/56adc916-d73a-4604-b371-1d5ea37b9f50-kube-api-access-95xdm\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:41.212838 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.212752 6107 generic.go:356] "Generic (PLEG): container finished" podID="5a4c7305-7cec-4351-aaf0-8c0cea11370d" containerID="a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e" exitCode=0 Oct 31 20:31:41.212838 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.212810 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" Oct 31 20:31:41.213120 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.212840 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" event={"ID":"5a4c7305-7cec-4351-aaf0-8c0cea11370d","Type":"ContainerDied","Data":"a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e"} Oct 31 20:31:41.213120 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.212931 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb" event={"ID":"5a4c7305-7cec-4351-aaf0-8c0cea11370d","Type":"ContainerDied","Data":"7f94df6e54a10af39f57040f196cb2e8997f0aceb9d6c796ea9c379929cb843b"} Oct 31 20:31:41.213120 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.212962 6107 scope.go:122] "RemoveContainer" containerID="a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e" Oct 31 20:31:41.214208 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.214185 6107 generic.go:356] "Generic (PLEG): container finished" podID="56adc916-d73a-4604-b371-1d5ea37b9f50" containerID="e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2" exitCode=0 Oct 31 20:31:41.214294 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.214226 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" Oct 31 20:31:41.214294 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.214238 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" event={"ID":"56adc916-d73a-4604-b371-1d5ea37b9f50","Type":"ContainerDied","Data":"e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2"} Oct 31 20:31:41.214294 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.214277 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9" event={"ID":"56adc916-d73a-4604-b371-1d5ea37b9f50","Type":"ContainerDied","Data":"2995e3b0ff8fd9bb09591c1977380134c811dce47c6fb8db6ac21ec9ff0f8073"} Oct 31 20:31:41.222354 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.222335 6107 scope.go:122] "RemoveContainer" containerID="a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e" Oct 31 20:31:41.222656 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:31:41.222638 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e\": container with ID starting with a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e not found: ID does not exist" containerID="a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e" Oct 31 20:31:41.222731 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.222669 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e"} err="failed to get container status \"a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e\": rpc error: code = NotFound desc = could not find container \"a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e\": container with ID starting with a1df8699af400d689958ca79ac1aa3cbd123b999f8467a183acf484e5ac63f9e not found: ID does not exist" Oct 31 20:31:41.222731 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.222692 6107 scope.go:122] "RemoveContainer" containerID="e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2" Oct 31 20:31:41.230943 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.230923 6107 scope.go:122] "RemoveContainer" containerID="e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2" Oct 31 20:31:41.231248 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:31:41.231228 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2\": container with ID starting with e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2 not found: ID does not exist" containerID="e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2" Oct 31 20:31:41.231320 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.231259 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2"} err="failed to get container status \"e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2\": rpc error: code = NotFound desc = could not find container \"e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2\": container with ID starting with e9baaf5af3e21df430a2e65083686acca4afcf3f351f1a5a693af7d9bc40f0f2 not found: ID does not exist" Oct 31 20:31:41.365479 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.365446 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb"] Oct 31 20:31:41.411401 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.411369 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-47884b3f-b5cc66fcf-226rb"] Oct 31 20:31:41.471135 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.471099 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9"] Oct 31 20:31:41.472779 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:41.472759 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-47884b3f-664cd55b9f-2nxn9"] Oct 31 20:31:42.486819 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:42.486788 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56adc916-d73a-4604-b371-1d5ea37b9f50" path="/var/lib/kubelet/pods/56adc916-d73a-4604-b371-1d5ea37b9f50/volumes" Oct 31 20:31:42.487234 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:42.487220 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4c7305-7cec-4351-aaf0-8c0cea11370d" path="/var/lib/kubelet/pods/5a4c7305-7cec-4351-aaf0-8c0cea11370d/volumes" Oct 31 20:31:46.435254 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.435215 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t"] Oct 31 20:31:46.435797 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.435774 6107 cpu_manager.go:414] "containerMap: removing container" podUID="373c6bd5-cd00-43f7-b515-7aa8606a2bc6" containerName="maas-api" Oct 31 20:31:46.435870 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.435800 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373c6bd5-cd00-43f7-b515-7aa8606a2bc6" containerName="maas-api" Oct 31 20:31:46.435870 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.435825 6107 cpu_manager.go:414] "containerMap: removing container" podUID="56adc916-d73a-4604-b371-1d5ea37b9f50" containerName="payload-pre-processing" Oct 31 20:31:46.435870 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.435833 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56adc916-d73a-4604-b371-1d5ea37b9f50" containerName="payload-pre-processing" Oct 31 20:31:46.436071 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.435869 6107 cpu_manager.go:414] "containerMap: removing container" podUID="5a4c7305-7cec-4351-aaf0-8c0cea11370d" containerName="payload-processing" Oct 31 20:31:46.436071 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.435877 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4c7305-7cec-4351-aaf0-8c0cea11370d" containerName="payload-processing" Oct 31 20:31:46.436071 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.436011 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4c7305-7cec-4351-aaf0-8c0cea11370d" containerName="payload-processing" Oct 31 20:31:46.436071 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.436030 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="373c6bd5-cd00-43f7-b515-7aa8606a2bc6" containerName="maas-api" Oct 31 20:31:46.436071 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.436042 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="56adc916-d73a-4604-b371-1d5ea37b9f50" containerName="payload-pre-processing" Oct 31 20:31:46.440655 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.440633 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.503806 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.503776 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-12dd5c-openshift-default-dockercfg-z42kh\"" Oct 31 20:31:46.537088 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.537056 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t"] Oct 31 20:31:46.580325 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.580285 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-credential-socket\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.580517 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.580332 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f50d96-c633-4f8f-aabd-54a65cc73037-istiod-ca-cert\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.580517 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.580469 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-envoy\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.580603 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.580518 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs4nc\" (UniqueName: \"kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-kube-api-access-rs4nc\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.580666 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.580641 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-certs\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.580716 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.580699 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-podinfo\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.580754 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.580729 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-token\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.580787 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.580778 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-socket\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.580842 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.580831 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-data\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.682575 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-credential-socket\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.682629 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f50d96-c633-4f8f-aabd-54a65cc73037-istiod-ca-cert\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.682681 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-envoy\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.682719 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs4nc\" (UniqueName: \"kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-kube-api-access-rs4nc\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.682765 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-certs\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.682803 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-podinfo\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.682826 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-token\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.682854 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-socket\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.682922 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-data\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.683020 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-credential-socket\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.683284 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-data\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.683288 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-certs\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.683753 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-socket\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.684129 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f50d96-c633-4f8f-aabd-54a65cc73037-istiod-ca-cert\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.686180 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-envoy\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.686965 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.686362 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-podinfo\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.735969 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.735937 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs4nc\" (UniqueName: \"kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-kube-api-access-rs4nc\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.736162 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.736069 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-token\") pod \"e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:46.754397 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:46.754366 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:47.163735 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:47.163686 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t"] Oct 31 20:31:47.166686 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:31:47.166658 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f50d96_c633_4f8f_aabd_54a65cc73037.slice/crio-6018a18bf8233d1f7fc2f5962ff6ba66f8fa4ce8d164b7ca22b49a4ed4f6241c WatchSource:0}: Error finding container 6018a18bf8233d1f7fc2f5962ff6ba66f8fa4ce8d164b7ca22b49a4ed4f6241c: Status 404 returned error can't find the container with id 6018a18bf8233d1f7fc2f5962ff6ba66f8fa4ce8d164b7ca22b49a4ed4f6241c Oct 31 20:31:47.168729 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:47.168694 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:31:47.168833 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:47.168774 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:31:47.168833 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:47.168816 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:31:47.253818 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:47.253776 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" event={"ID":"23f50d96-c633-4f8f-aabd-54a65cc73037","Type":"ContainerStarted","Data":"8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132"} Oct 31 20:31:47.254268 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:47.253841 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" event={"ID":"23f50d96-c633-4f8f-aabd-54a65cc73037","Type":"ContainerStarted","Data":"6018a18bf8233d1f7fc2f5962ff6ba66f8fa4ce8d164b7ca22b49a4ed4f6241c"} Oct 31 20:31:47.363661 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:47.363601 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" podStartSLOduration=1.363586281 podStartE2EDuration="1.363586281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:31:47.360756898 +0000 UTC m=+4478.994150806" watchObservedRunningTime="2026-10-31 20:31:47.363586281 +0000 UTC m=+4478.996980240" Oct 31 20:31:47.754707 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:47.754665 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:48.760455 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:48.760420 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:49.264567 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:49.264543 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:49.265524 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:49.265495 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:31:50.406988 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:31:50.406950 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:31:53.749173 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.749137 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq"] Oct 31 20:31:53.753366 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.753340 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" Oct 31 20:31:53.830275 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.830244 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-12dd5c\"" Oct 31 20:31:53.833695 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.833672 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq"] Oct 31 20:31:53.850864 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.850816 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-12dd5c-dockercfg-pk2cl\"" Oct 31 20:31:53.858045 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.858013 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a93ba5-5277-41b7-b52f-e7fdecaa737e-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq\" (UID: \"49a93ba5-5277-41b7-b52f-e7fdecaa737e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" Oct 31 20:31:53.858181 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.858053 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25f8c\" (UniqueName: \"kubernetes.io/projected/49a93ba5-5277-41b7-b52f-e7fdecaa737e-kube-api-access-25f8c\") pod \"payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq\" (UID: \"49a93ba5-5277-41b7-b52f-e7fdecaa737e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" Oct 31 20:31:53.906621 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.906560 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp"] Oct 31 20:31:53.912969 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.912937 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" Oct 31 20:31:53.959632 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.959572 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a93ba5-5277-41b7-b52f-e7fdecaa737e-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq\" (UID: \"49a93ba5-5277-41b7-b52f-e7fdecaa737e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" Oct 31 20:31:53.959949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.959920 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25f8c\" (UniqueName: \"kubernetes.io/projected/49a93ba5-5277-41b7-b52f-e7fdecaa737e-kube-api-access-25f8c\") pod \"payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq\" (UID: \"49a93ba5-5277-41b7-b52f-e7fdecaa737e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" Oct 31 20:31:53.960450 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.960426 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a93ba5-5277-41b7-b52f-e7fdecaa737e-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq\" (UID: \"49a93ba5-5277-41b7-b52f-e7fdecaa737e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" Oct 31 20:31:53.961483 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:53.961457 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp"] Oct 31 20:31:54.013516 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:54.013433 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25f8c\" (UniqueName: \"kubernetes.io/projected/49a93ba5-5277-41b7-b52f-e7fdecaa737e-kube-api-access-25f8c\") pod \"payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq\" (UID: \"49a93ba5-5277-41b7-b52f-e7fdecaa737e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" Oct 31 20:31:54.060626 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:54.060591 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cgs6\" (UniqueName: \"kubernetes.io/projected/c09057e5-e1e0-4a81-ba68-b6311c0f1661-kube-api-access-9cgs6\") pod \"payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp\" (UID: \"c09057e5-e1e0-4a81-ba68-b6311c0f1661\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" Oct 31 20:31:54.060823 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:54.060650 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09057e5-e1e0-4a81-ba68-b6311c0f1661-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp\" (UID: \"c09057e5-e1e0-4a81-ba68-b6311c0f1661\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" Oct 31 20:31:54.067561 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:54.067536 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" Oct 31 20:31:54.161781 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:54.161745 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cgs6\" (UniqueName: \"kubernetes.io/projected/c09057e5-e1e0-4a81-ba68-b6311c0f1661-kube-api-access-9cgs6\") pod \"payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp\" (UID: \"c09057e5-e1e0-4a81-ba68-b6311c0f1661\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" Oct 31 20:31:54.162007 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:54.161922 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09057e5-e1e0-4a81-ba68-b6311c0f1661-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp\" (UID: \"c09057e5-e1e0-4a81-ba68-b6311c0f1661\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" Oct 31 20:31:54.168900 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:54.163181 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09057e5-e1e0-4a81-ba68-b6311c0f1661-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp\" (UID: \"c09057e5-e1e0-4a81-ba68-b6311c0f1661\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" Oct 31 20:31:54.220397 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:54.220356 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cgs6\" (UniqueName: \"kubernetes.io/projected/c09057e5-e1e0-4a81-ba68-b6311c0f1661-kube-api-access-9cgs6\") pod \"payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp\" (UID: \"c09057e5-e1e0-4a81-ba68-b6311c0f1661\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" Oct 31 20:31:54.230264 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:54.230234 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" Oct 31 20:31:54.432482 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:54.432407 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq"] Oct 31 20:31:54.435065 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:31:54.435036 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a93ba5_5277_41b7_b52f_e7fdecaa737e.slice/crio-486ead87dce5903bf40fbd56a16d5530242d4f3b2f879b6af3eabf4643493850 WatchSource:0}: Error finding container 486ead87dce5903bf40fbd56a16d5530242d4f3b2f879b6af3eabf4643493850: Status 404 returned error can't find the container with id 486ead87dce5903bf40fbd56a16d5530242d4f3b2f879b6af3eabf4643493850 Oct 31 20:31:54.594156 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:54.594125 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp"] Oct 31 20:31:54.596451 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:31:54.596423 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09057e5_e1e0_4a81_ba68_b6311c0f1661.slice/crio-f5d179a7d166c2a400c8bc07f3c28605053b72fa79674512a8a03befe3745a1d WatchSource:0}: Error finding container f5d179a7d166c2a400c8bc07f3c28605053b72fa79674512a8a03befe3745a1d: Status 404 returned error can't find the container with id f5d179a7d166c2a400c8bc07f3c28605053b72fa79674512a8a03befe3745a1d Oct 31 20:31:55.300830 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:55.300787 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" event={"ID":"49a93ba5-5277-41b7-b52f-e7fdecaa737e","Type":"ContainerStarted","Data":"2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b"} Oct 31 20:31:55.301317 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:55.300860 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" event={"ID":"49a93ba5-5277-41b7-b52f-e7fdecaa737e","Type":"ContainerStarted","Data":"486ead87dce5903bf40fbd56a16d5530242d4f3b2f879b6af3eabf4643493850"} Oct 31 20:31:55.301317 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:55.301015 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" Oct 31 20:31:55.302466 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:55.302442 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" event={"ID":"c09057e5-e1e0-4a81-ba68-b6311c0f1661","Type":"ContainerStarted","Data":"03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a"} Oct 31 20:31:55.302554 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:55.302479 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" event={"ID":"c09057e5-e1e0-4a81-ba68-b6311c0f1661","Type":"ContainerStarted","Data":"f5d179a7d166c2a400c8bc07f3c28605053b72fa79674512a8a03befe3745a1d"} Oct 31 20:31:55.302554 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:55.302500 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" Oct 31 20:31:55.412062 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:55.412010 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" podStartSLOduration=2.411995805 podStartE2EDuration="2.411995805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:31:55.408999626 +0000 UTC m=+4487.042393534" watchObservedRunningTime="2026-10-31 20:31:55.411995805 +0000 UTC m=+4487.045389712" Oct 31 20:31:55.515290 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:31:55.515227 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" podStartSLOduration=2.5152135209999997 podStartE2EDuration="2.515213521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:31:55.514901324 +0000 UTC m=+4487.148295227" watchObservedRunningTime="2026-10-31 20:31:55.515213521 +0000 UTC m=+4487.148607429" Oct 31 20:32:06.325770 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:06.325742 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" Oct 31 20:32:06.326470 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:06.326451 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" Oct 31 20:32:11.400356 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:11.400321 6107 generic.go:356] "Generic (PLEG): container finished" podID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerID="8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132" exitCode=0 Oct 31 20:32:11.400832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:11.400395 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" event={"ID":"23f50d96-c633-4f8f-aabd-54a65cc73037","Type":"ContainerDied","Data":"8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132"} Oct 31 20:32:11.400832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:11.400780 6107 scope.go:122] "RemoveContainer" containerID="8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132" Oct 31 20:32:11.402022 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:11.401996 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:32:11.402091 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:11.402061 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:32:11.402128 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:11.402093 6107 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 20:32:12.430905 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:12.430016 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" event={"ID":"23f50d96-c633-4f8f-aabd-54a65cc73037","Type":"ContainerStarted","Data":"a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d"} Oct 31 20:32:12.755530 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:12.755490 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:32:12.755729 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:12.755606 6107 patch_prober.go:29] interesting pod/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 20:32:12.755729 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:12.755658 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 20:32:13.754945 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:13.754913 6107 patch_prober.go:29] interesting pod/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 20:32:13.755344 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:13.754963 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 20:32:14.755209 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:14.755173 6107 patch_prober.go:29] interesting pod/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 20:32:14.755663 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:14.755232 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 20:32:15.754656 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:15.754621 6107 patch_prober.go:29] interesting pod/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 20:32:15.754833 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:15.754673 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 20:32:16.755286 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:16.755243 6107 patch_prober.go:29] interesting pod/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 20:32:16.755700 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:16.755314 6107 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 20:32:17.759720 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:17.759684 6107 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:32:17.760220 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:17.760082 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:32:17.760996 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:32:17.760970 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:32:20.403958 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:32:20.403921 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:32:50.402049 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:32:50.402012 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:33:20.402317 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:33:20.402278 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:33:44.284576 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:44.284537 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp"] Oct 31 20:33:44.390344 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:44.390312 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq"] Oct 31 20:33:49.284938 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.284852 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" podUID="c09057e5-e1e0-4a81-ba68-b6311c0f1661" containerName="payload-processing" containerID="cri-o://03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a" gracePeriod=25 Oct 31 20:33:49.391494 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.391455 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" podUID="49a93ba5-5277-41b7-b52f-e7fdecaa737e" containerName="payload-pre-processing" containerID="cri-o://2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b" gracePeriod=25 Oct 31 20:33:49.531788 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.531757 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" Oct 31 20:33:49.654633 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.654609 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" Oct 31 20:33:49.690551 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.690528 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c09057e5-e1e0-4a81-ba68-b6311c0f1661-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09057e5-e1e0-4a81-ba68-b6311c0f1661-plugins-config-volume\") pod \"c09057e5-e1e0-4a81-ba68-b6311c0f1661\" (UID: \"c09057e5-e1e0-4a81-ba68-b6311c0f1661\") " Oct 31 20:33:49.690681 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.690667 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c09057e5-e1e0-4a81-ba68-b6311c0f1661-kube-api-access-9cgs6\" (UniqueName: \"kubernetes.io/projected/c09057e5-e1e0-4a81-ba68-b6311c0f1661-kube-api-access-9cgs6\") pod \"c09057e5-e1e0-4a81-ba68-b6311c0f1661\" (UID: \"c09057e5-e1e0-4a81-ba68-b6311c0f1661\") " Oct 31 20:33:49.690860 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.690841 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c09057e5-e1e0-4a81-ba68-b6311c0f1661-plugins-config-volume" pod "c09057e5-e1e0-4a81-ba68-b6311c0f1661" (UID: "c09057e5-e1e0-4a81-ba68-b6311c0f1661"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:33:49.690969 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.690957 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c09057e5-e1e0-4a81-ba68-b6311c0f1661-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:33:49.692675 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.692655 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09057e5-e1e0-4a81-ba68-b6311c0f1661-kube-api-access-9cgs6" pod "c09057e5-e1e0-4a81-ba68-b6311c0f1661" (UID: "c09057e5-e1e0-4a81-ba68-b6311c0f1661"). InnerVolumeSpecName "kube-api-access-9cgs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:33:49.791360 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.791325 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a93ba5-5277-41b7-b52f-e7fdecaa737e-kube-api-access-25f8c\" (UniqueName: \"kubernetes.io/projected/49a93ba5-5277-41b7-b52f-e7fdecaa737e-kube-api-access-25f8c\") pod \"49a93ba5-5277-41b7-b52f-e7fdecaa737e\" (UID: \"49a93ba5-5277-41b7-b52f-e7fdecaa737e\") " Oct 31 20:33:49.791548 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.791392 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a93ba5-5277-41b7-b52f-e7fdecaa737e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a93ba5-5277-41b7-b52f-e7fdecaa737e-plugins-config-volume\") pod \"49a93ba5-5277-41b7-b52f-e7fdecaa737e\" (UID: \"49a93ba5-5277-41b7-b52f-e7fdecaa737e\") " Oct 31 20:33:49.791704 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.791685 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cgs6\" (UniqueName: \"kubernetes.io/projected/c09057e5-e1e0-4a81-ba68-b6311c0f1661-kube-api-access-9cgs6\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:33:49.791770 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.791736 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a93ba5-5277-41b7-b52f-e7fdecaa737e-plugins-config-volume" pod "49a93ba5-5277-41b7-b52f-e7fdecaa737e" (UID: "49a93ba5-5277-41b7-b52f-e7fdecaa737e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:33:49.793333 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.793314 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a93ba5-5277-41b7-b52f-e7fdecaa737e-kube-api-access-25f8c" pod "49a93ba5-5277-41b7-b52f-e7fdecaa737e" (UID: "49a93ba5-5277-41b7-b52f-e7fdecaa737e"). InnerVolumeSpecName "kube-api-access-25f8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:33:49.892423 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.892345 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25f8c\" (UniqueName: \"kubernetes.io/projected/49a93ba5-5277-41b7-b52f-e7fdecaa737e-kube-api-access-25f8c\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:33:49.892423 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.892372 6107 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a93ba5-5277-41b7-b52f-e7fdecaa737e-plugins-config-volume\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:33:49.919311 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.919278 6107 generic.go:356] "Generic (PLEG): container finished" podID="c09057e5-e1e0-4a81-ba68-b6311c0f1661" containerID="03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a" exitCode=0 Oct 31 20:33:49.919477 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.919330 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" Oct 31 20:33:49.919477 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.919350 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" event={"ID":"c09057e5-e1e0-4a81-ba68-b6311c0f1661","Type":"ContainerDied","Data":"03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a"} Oct 31 20:33:49.919477 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.919421 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp" event={"ID":"c09057e5-e1e0-4a81-ba68-b6311c0f1661","Type":"ContainerDied","Data":"f5d179a7d166c2a400c8bc07f3c28605053b72fa79674512a8a03befe3745a1d"} Oct 31 20:33:49.919477 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.919445 6107 scope.go:122] "RemoveContainer" containerID="03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a" Oct 31 20:33:49.920747 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.920722 6107 generic.go:356] "Generic (PLEG): container finished" podID="49a93ba5-5277-41b7-b52f-e7fdecaa737e" containerID="2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b" exitCode=0 Oct 31 20:33:49.920907 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.920774 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" Oct 31 20:33:49.920907 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.920794 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" event={"ID":"49a93ba5-5277-41b7-b52f-e7fdecaa737e","Type":"ContainerDied","Data":"2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b"} Oct 31 20:33:49.920907 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.920834 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq" event={"ID":"49a93ba5-5277-41b7-b52f-e7fdecaa737e","Type":"ContainerDied","Data":"486ead87dce5903bf40fbd56a16d5530242d4f3b2f879b6af3eabf4643493850"} Oct 31 20:33:49.928847 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.928828 6107 scope.go:122] "RemoveContainer" containerID="03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a" Oct 31 20:33:49.929173 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:33:49.929149 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a\": container with ID starting with 03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a not found: ID does not exist" containerID="03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a" Oct 31 20:33:49.929293 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.929177 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a"} err="failed to get container status \"03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a\": rpc error: code = NotFound desc = could not find container \"03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a\": container with ID starting with 03a6e4188f5b75ac4b3e0f180b616869109f2cf7fc9a6508720a82bc31e2d89a not found: ID does not exist" Oct 31 20:33:49.929293 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.929192 6107 scope.go:122] "RemoveContainer" containerID="2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b" Oct 31 20:33:49.937139 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.937124 6107 scope.go:122] "RemoveContainer" containerID="2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b" Oct 31 20:33:49.937392 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:33:49.937375 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b\": container with ID starting with 2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b not found: ID does not exist" containerID="2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b" Oct 31 20:33:49.937436 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:49.937398 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b"} err="failed to get container status \"2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b\": rpc error: code = NotFound desc = could not find container \"2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b\": container with ID starting with 2ffb89c5cae824f77eae2fbaefe64c8198f3c3bb58e690c3b6390d26b7b7541b not found: ID does not exist" Oct 31 20:33:50.071056 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:50.071020 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp"] Oct 31 20:33:50.077029 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:50.077003 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-12dd5c-567848cfb9-m9rpp"] Oct 31 20:33:50.177137 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:50.177061 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq"] Oct 31 20:33:50.182694 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:50.182667 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-12dd5c-c54994df5-9nwqq"] Oct 31 20:33:50.402270 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:33:50.402238 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:33:50.485274 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:50.485245 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a93ba5-5277-41b7-b52f-e7fdecaa737e" path="/var/lib/kubelet/pods/49a93ba5-5277-41b7-b52f-e7fdecaa737e/volumes" Oct 31 20:33:50.485767 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:33:50.485750 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c09057e5-e1e0-4a81-ba68-b6311c0f1661" path="/var/lib/kubelet/pods/c09057e5-e1e0-4a81-ba68-b6311c0f1661/volumes" Oct 31 20:34:05.261898 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:05.261852 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t"] Oct 31 20:34:05.262309 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:05.262119 6107 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerName="istio-proxy" containerID="cri-o://a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d" gracePeriod=30 Oct 31 20:34:10.504271 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.504244 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:34:10.590707 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.590674 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f50d96-c633-4f8f-aabd-54a65cc73037-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f50d96-c633-4f8f-aabd-54a65cc73037-istiod-ca-cert\") pod \"23f50d96-c633-4f8f-aabd-54a65cc73037\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " Oct 31 20:34:10.590904 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.590724 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-data\") pod \"23f50d96-c633-4f8f-aabd-54a65cc73037\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " Oct 31 20:34:10.590904 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.590776 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-socket\") pod \"23f50d96-c633-4f8f-aabd-54a65cc73037\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " Oct 31 20:34:10.590904 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.590807 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-envoy\") pod \"23f50d96-c633-4f8f-aabd-54a65cc73037\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " Oct 31 20:34:10.590904 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.590834 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-certs\") pod \"23f50d96-c633-4f8f-aabd-54a65cc73037\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " Oct 31 20:34:10.591114 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.590924 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-token\" (UniqueName: \"kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-token\") pod \"23f50d96-c633-4f8f-aabd-54a65cc73037\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " Oct 31 20:34:10.591114 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.590999 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-credential-socket\") pod \"23f50d96-c633-4f8f-aabd-54a65cc73037\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " Oct 31 20:34:10.591114 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.591008 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-data" pod "23f50d96-c633-4f8f-aabd-54a65cc73037" (UID: "23f50d96-c633-4f8f-aabd-54a65cc73037"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:34:10.591114 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.591088 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-podinfo\") pod \"23f50d96-c633-4f8f-aabd-54a65cc73037\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " Oct 31 20:34:10.591114 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.591103 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-socket" pod "23f50d96-c633-4f8f-aabd-54a65cc73037" (UID: "23f50d96-c633-4f8f-aabd-54a65cc73037"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:34:10.591372 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.591126 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f50d96-c633-4f8f-aabd-54a65cc73037-istiod-ca-cert" pod "23f50d96-c633-4f8f-aabd-54a65cc73037" (UID: "23f50d96-c633-4f8f-aabd-54a65cc73037"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:34:10.591372 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.591152 6107 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-kube-api-access-rs4nc\" (UniqueName: \"kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-kube-api-access-rs4nc\") pod \"23f50d96-c633-4f8f-aabd-54a65cc73037\" (UID: \"23f50d96-c633-4f8f-aabd-54a65cc73037\") " Oct 31 20:34:10.591496 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.591409 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-credential-socket" pod "23f50d96-c633-4f8f-aabd-54a65cc73037" (UID: "23f50d96-c633-4f8f-aabd-54a65cc73037"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:34:10.591496 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.591469 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-certs" pod "23f50d96-c633-4f8f-aabd-54a65cc73037" (UID: "23f50d96-c633-4f8f-aabd-54a65cc73037"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:34:10.591735 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.591712 6107 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-data\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:34:10.591963 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.591945 6107 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:34:10.592108 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.592085 6107 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-workload-certs\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:34:10.592198 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.592111 6107 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-credential-socket\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:34:10.592198 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.592124 6107 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f50d96-c633-4f8f-aabd-54a65cc73037-istiod-ca-cert\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:34:10.593191 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.593172 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-envoy" pod "23f50d96-c633-4f8f-aabd-54a65cc73037" (UID: "23f50d96-c633-4f8f-aabd-54a65cc73037"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:34:10.593495 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.593470 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-kube-api-access-rs4nc" pod "23f50d96-c633-4f8f-aabd-54a65cc73037" (UID: "23f50d96-c633-4f8f-aabd-54a65cc73037"). InnerVolumeSpecName "kube-api-access-rs4nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:34:10.593602 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.593531 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-token" pod "23f50d96-c633-4f8f-aabd-54a65cc73037" (UID: "23f50d96-c633-4f8f-aabd-54a65cc73037"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:34:10.593602 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.593544 6107 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-podinfo" pod "23f50d96-c633-4f8f-aabd-54a65cc73037" (UID: "23f50d96-c633-4f8f-aabd-54a65cc73037"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:34:10.693282 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.693206 6107 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-podinfo\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:34:10.693282 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.693233 6107 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs4nc\" (UniqueName: \"kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-kube-api-access-rs4nc\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:34:10.693282 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.693244 6107 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-envoy\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:34:10.693282 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:10.693252 6107 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f50d96-c633-4f8f-aabd-54a65cc73037-istio-token\") on node \"ip-10-0-128-164.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:34:11.030671 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:11.030629 6107 generic.go:356] "Generic (PLEG): container finished" podID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerID="a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d" exitCode=0 Oct 31 20:34:11.030860 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:11.030680 6107 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" Oct 31 20:34:11.030860 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:11.030714 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" event={"ID":"23f50d96-c633-4f8f-aabd-54a65cc73037","Type":"ContainerDied","Data":"a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d"} Oct 31 20:34:11.030860 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:11.030780 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t" event={"ID":"23f50d96-c633-4f8f-aabd-54a65cc73037","Type":"ContainerDied","Data":"6018a18bf8233d1f7fc2f5962ff6ba66f8fa4ce8d164b7ca22b49a4ed4f6241c"} Oct 31 20:34:11.030860 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:11.030843 6107 scope.go:122] "RemoveContainer" containerID="a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d" Oct 31 20:34:11.041600 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:11.041579 6107 scope.go:122] "RemoveContainer" containerID="8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132" Oct 31 20:34:11.049696 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:11.049677 6107 scope.go:122] "RemoveContainer" containerID="a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d" Oct 31 20:34:11.050066 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:34:11.050048 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d\": container with ID starting with a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d not found: ID does not exist" containerID="a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d" Oct 31 20:34:11.050147 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:11.050078 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d"} err="failed to get container status \"a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d\": rpc error: code = NotFound desc = could not find container \"a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d\": container with ID starting with a52cfcff3bcfef83bf7a0c3aee2dc6d9086eb1c6502ae07f6e2ab41688546b6d not found: ID does not exist" Oct 31 20:34:11.050147 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:11.050097 6107 scope.go:122] "RemoveContainer" containerID="8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132" Oct 31 20:34:11.050323 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:34:11.050309 6107 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132\": container with ID starting with 8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132 not found: ID does not exist" containerID="8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132" Oct 31 20:34:11.050365 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:11.050327 6107 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132"} err="failed to get container status \"8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132\": rpc error: code = NotFound desc = could not find container \"8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132\": container with ID starting with 8ffe17ce7754643657e628cf5cfb6d08f9c4ffa01b5fe11c798d4c8ca5212132 not found: ID does not exist" Oct 31 20:34:11.183150 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:11.183118 6107 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t"] Oct 31 20:34:11.186062 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:11.186036 6107 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-12dd5c-openshift-default-77b6db784f-54d5t"] Oct 31 20:34:12.488897 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:34:12.488852 6107 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" path="/var/lib/kubelet/pods/23f50d96-c633-4f8f-aabd-54a65cc73037/volumes" Oct 31 20:34:20.401950 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:34:20.401912 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:34:50.406916 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:34:50.406816 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:35:20.406978 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:35:20.406861 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:35:27.455455 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:27.455423 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8584f67695-vgddt_2b9d3a1a-82a2-491a-a272-0275351a3281/manager/0.log" Oct 31 20:35:30.769318 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:30.769285 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-74f5c976df-bl8kh_c83c38f0-bc64-4713-9299-f8251aa4333a/maas-api/0.log" Oct 31 20:35:34.536223 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:34.536186 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-89l6r_7ad70bd3-5d6a-4631-b5fe-e555d708bb44/manager/0.log" Oct 31 20:35:36.498178 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:36.498122 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b655b5cf-k44r9_a79481f4-b755-490e-8717-b542362b80ad/kube-auth-proxy/0.log" Oct 31 20:35:37.618757 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:37.618725 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-jkrrj_dca75e6d-22eb-42ed-9d6c-4edb0d5678ef/istio-proxy/0.log" Oct 31 20:35:38.109515 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:38.109482 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-r67h6_7fb2c48a-65ce-436f-8e39-29d5f7aa0969/istio-proxy/0.log" Oct 31 20:35:39.012846 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:39.012806 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-zjsq8_a5360256-7648-404e-ae6c-e5556cca1ae3/istio-proxy/0.log" Oct 31 20:35:40.984972 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:40.984944 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-r5tcm_b0d359d1-caca-4b2d-9a96-a7ce1363a407/main/0.log" Oct 31 20:35:41.443764 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:41.443689 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-shbmz_a78e6484-6c0b-4c24-a693-1b7c55fe70f3/main/0.log" Oct 31 20:35:41.884562 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:41.884533 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-6m88h_3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278/main/0.log" Oct 31 20:35:42.320246 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:42.320221 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v_5019f17d-138f-4f54-b963-51645b530ffe/main/0.log" Oct 31 20:35:50.402618 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:35:50.402580 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:35:52.272935 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:52.272878 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8584f67695-vgddt_2b9d3a1a-82a2-491a-a272-0275351a3281/manager/0.log" Oct 31 20:35:54.919188 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:54.919141 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-74f5c976df-bl8kh_c83c38f0-bc64-4713-9299-f8251aa4333a/maas-api/0.log" Oct 31 20:35:58.243560 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:35:58.243524 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-89l6r_7ad70bd3-5d6a-4631-b5fe-e555d708bb44/manager/0.log" Oct 31 20:36:00.015298 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:36:00.015266 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b655b5cf-k44r9_a79481f4-b755-490e-8717-b542362b80ad/kube-auth-proxy/0.log" Oct 31 20:36:01.152365 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:36:01.152330 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-jkrrj_dca75e6d-22eb-42ed-9d6c-4edb0d5678ef/istio-proxy/0.log" Oct 31 20:36:01.600532 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:36:01.600500 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-r67h6_7fb2c48a-65ce-436f-8e39-29d5f7aa0969/istio-proxy/0.log" Oct 31 20:36:02.534051 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:36:02.534016 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-zjsq8_a5360256-7648-404e-ae6c-e5556cca1ae3/istio-proxy/0.log" Oct 31 20:36:04.278686 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:36:04.278655 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-r5tcm_b0d359d1-caca-4b2d-9a96-a7ce1363a407/main/0.log" Oct 31 20:36:04.762119 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:36:04.762081 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-shbmz_a78e6484-6c0b-4c24-a693-1b7c55fe70f3/main/0.log" Oct 31 20:36:05.199185 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:36:05.199089 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-6m88h_3be4fac3-79cf-4dfc-bf3b-6ff4f8d60278/main/0.log" Oct 31 20:36:05.634758 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:36:05.634726 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-t5b6v_5019f17d-138f-4f54-b963-51645b530ffe/main/0.log" Oct 31 20:36:20.403130 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:36:20.403092 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:36:50.403084 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:36:50.402997 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:37:13.862484 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.862453 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ks7ld/must-gather-7rg5g"] Oct 31 20:37:13.864832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.862827 6107 cpu_manager.go:414] "containerMap: removing container" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerName="istio-proxy" Oct 31 20:37:13.864832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.862840 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerName="istio-proxy" Oct 31 20:37:13.864832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.862857 6107 cpu_manager.go:414] "containerMap: removing container" podUID="49a93ba5-5277-41b7-b52f-e7fdecaa737e" containerName="payload-pre-processing" Oct 31 20:37:13.864832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.862865 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a93ba5-5277-41b7-b52f-e7fdecaa737e" containerName="payload-pre-processing" Oct 31 20:37:13.864832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.862894 6107 cpu_manager.go:414] "containerMap: removing container" podUID="c09057e5-e1e0-4a81-ba68-b6311c0f1661" containerName="payload-processing" Oct 31 20:37:13.864832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.862902 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c09057e5-e1e0-4a81-ba68-b6311c0f1661" containerName="payload-processing" Oct 31 20:37:13.864832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.862942 6107 cpu_manager.go:414] "containerMap: removing container" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerName="istio-proxy" Oct 31 20:37:13.864832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.862950 6107 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerName="istio-proxy" Oct 31 20:37:13.864832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.863024 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerName="istio-proxy" Oct 31 20:37:13.864832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.863036 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a93ba5-5277-41b7-b52f-e7fdecaa737e" containerName="payload-pre-processing" Oct 31 20:37:13.864832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.863045 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="c09057e5-e1e0-4a81-ba68-b6311c0f1661" containerName="payload-processing" Oct 31 20:37:13.864832 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.863161 6107 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f50d96-c633-4f8f-aabd-54a65cc73037" containerName="istio-proxy" Oct 31 20:37:13.865964 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.865948 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ks7ld/must-gather-7rg5g" Oct 31 20:37:13.960333 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.960303 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ks7ld\"/\"openshift-service-ca.crt\"" Oct 31 20:37:13.960495 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.960382 6107 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ks7ld\"/\"default-dockercfg-lbbt2\"" Oct 31 20:37:13.960495 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.960386 6107 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ks7ld\"/\"kube-root-ca.crt\"" Oct 31 20:37:13.962714 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.962684 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ks7ld/must-gather-7rg5g"] Oct 31 20:37:13.991295 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.991269 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wl9\" (UniqueName: \"kubernetes.io/projected/34aac6b1-d1aa-4f93-8c7a-309053af00a9-kube-api-access-85wl9\") pod \"must-gather-7rg5g\" (UID: \"34aac6b1-d1aa-4f93-8c7a-309053af00a9\") " pod="openshift-must-gather-ks7ld/must-gather-7rg5g" Oct 31 20:37:13.991428 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:13.991306 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34aac6b1-d1aa-4f93-8c7a-309053af00a9-must-gather-output\") pod \"must-gather-7rg5g\" (UID: \"34aac6b1-d1aa-4f93-8c7a-309053af00a9\") " pod="openshift-must-gather-ks7ld/must-gather-7rg5g" Oct 31 20:37:14.092767 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:14.092733 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34aac6b1-d1aa-4f93-8c7a-309053af00a9-must-gather-output\") pod \"must-gather-7rg5g\" (UID: \"34aac6b1-d1aa-4f93-8c7a-309053af00a9\") " pod="openshift-must-gather-ks7ld/must-gather-7rg5g" Oct 31 20:37:14.092971 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:14.092903 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wl9\" (UniqueName: \"kubernetes.io/projected/34aac6b1-d1aa-4f93-8c7a-309053af00a9-kube-api-access-85wl9\") pod \"must-gather-7rg5g\" (UID: \"34aac6b1-d1aa-4f93-8c7a-309053af00a9\") " pod="openshift-must-gather-ks7ld/must-gather-7rg5g" Oct 31 20:37:14.093153 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:14.093133 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34aac6b1-d1aa-4f93-8c7a-309053af00a9-must-gather-output\") pod \"must-gather-7rg5g\" (UID: \"34aac6b1-d1aa-4f93-8c7a-309053af00a9\") " pod="openshift-must-gather-ks7ld/must-gather-7rg5g" Oct 31 20:37:14.177460 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:14.177390 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wl9\" (UniqueName: \"kubernetes.io/projected/34aac6b1-d1aa-4f93-8c7a-309053af00a9-kube-api-access-85wl9\") pod \"must-gather-7rg5g\" (UID: \"34aac6b1-d1aa-4f93-8c7a-309053af00a9\") " pod="openshift-must-gather-ks7ld/must-gather-7rg5g" Oct 31 20:37:14.179303 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:14.179285 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ks7ld/must-gather-7rg5g" Oct 31 20:37:14.538608 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:14.538582 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ks7ld/must-gather-7rg5g"] Oct 31 20:37:14.541112 ip-10-0-128-164 kubenswrapper[6107]: W1031 20:37:14.541081 6107 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34aac6b1_d1aa_4f93_8c7a_309053af00a9.slice/crio-e37710910c21f8b4cae8a893e1eddc051cb6b694e0b1ec8a3327a802698ab360 WatchSource:0}: Error finding container e37710910c21f8b4cae8a893e1eddc051cb6b694e0b1ec8a3327a802698ab360: Status 404 returned error can't find the container with id e37710910c21f8b4cae8a893e1eddc051cb6b694e0b1ec8a3327a802698ab360 Oct 31 20:37:14.542949 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:14.542928 6107 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 20:37:15.013324 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:15.013287 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ks7ld/must-gather-7rg5g" event={"ID":"34aac6b1-d1aa-4f93-8c7a-309053af00a9","Type":"ContainerStarted","Data":"e37710910c21f8b4cae8a893e1eddc051cb6b694e0b1ec8a3327a802698ab360"} Oct 31 20:37:17.026937 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:17.026898 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ks7ld/must-gather-7rg5g" event={"ID":"34aac6b1-d1aa-4f93-8c7a-309053af00a9","Type":"ContainerStarted","Data":"bec8a01a4256170156fdbe252ff8f71308cc95708218fb9637394631b0cf7e39"} Oct 31 20:37:17.027427 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:17.026949 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ks7ld/must-gather-7rg5g" event={"ID":"34aac6b1-d1aa-4f93-8c7a-309053af00a9","Type":"ContainerStarted","Data":"e67f72baca7d7ea6e976f4fe209991271f4372a04567f957b1fc0105f4d096ca"} Oct 31 20:37:17.132756 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:17.132693 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-ks7ld/must-gather-7rg5g" podStartSLOduration=2.5936316010000002 podStartE2EDuration="4.132675343s" totalImagesPullingTime="1.539043742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:37:17.131804257 +0000 UTC m=+4808.765198163" watchObservedRunningTime="2026-10-31 20:37:17.132675343 +0000 UTC m=+4808.766069250" Oct 31 20:37:20.405269 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:37:20.405233 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor" Oct 31 20:37:23.514354 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:23.514315 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9hfm2_f947add9-2413-4c26-860a-a5431e7cd2b9/global-pull-secret-syncer/0.log" Oct 31 20:37:27.309426 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:27.309395 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-szn4c_474d47d1-f185-4127-a085-786c6ada462e/konnectivity-agent/0.log" Oct 31 20:37:28.094182 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:28.094152 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-164.us-west-2.compute.internal_3d2e8e59ab02a8ff64cf5b536f467713/haproxy/0.log" Oct 31 20:37:32.704230 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.704191 6107 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz"] Oct 31 20:37:32.712899 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.712821 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.784435 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.784404 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bkl\" (UniqueName: \"kubernetes.io/projected/b6dc6fd1-1e89-4330-bb79-881ad9f96585-kube-api-access-t2bkl\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.784602 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.784442 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6dc6fd1-1e89-4330-bb79-881ad9f96585-podres\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.784602 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.784536 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6dc6fd1-1e89-4330-bb79-881ad9f96585-sys\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.784602 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.784588 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6dc6fd1-1e89-4330-bb79-881ad9f96585-lib-modules\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.784775 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.784702 6107 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6dc6fd1-1e89-4330-bb79-881ad9f96585-proc\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.790045 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.790016 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz"] Oct 31 20:37:32.885388 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.885356 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bkl\" (UniqueName: \"kubernetes.io/projected/b6dc6fd1-1e89-4330-bb79-881ad9f96585-kube-api-access-t2bkl\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.885388 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.885394 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6dc6fd1-1e89-4330-bb79-881ad9f96585-podres\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.885646 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.885434 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6dc6fd1-1e89-4330-bb79-881ad9f96585-sys\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.885646 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.885507 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6dc6fd1-1e89-4330-bb79-881ad9f96585-lib-modules\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.885646 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.885551 6107 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6dc6fd1-1e89-4330-bb79-881ad9f96585-proc\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.885646 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.885563 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6dc6fd1-1e89-4330-bb79-881ad9f96585-podres\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.885646 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.885600 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6dc6fd1-1e89-4330-bb79-881ad9f96585-lib-modules\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.885646 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.885523 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6dc6fd1-1e89-4330-bb79-881ad9f96585-sys\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.885646 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.885634 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6dc6fd1-1e89-4330-bb79-881ad9f96585-proc\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:32.938629 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:32.938596 6107 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bkl\" (UniqueName: \"kubernetes.io/projected/b6dc6fd1-1e89-4330-bb79-881ad9f96585-kube-api-access-t2bkl\") pod \"perf-node-gather-daemonset-kgmsz\" (UID: \"b6dc6fd1-1e89-4330-bb79-881ad9f96585\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:33.026584 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:33.026553 6107 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:33.397407 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:33.397369 6107 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz"] Oct 31 20:37:33.431321 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:33.431296 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-89l6r_7ad70bd3-5d6a-4631-b5fe-e555d708bb44/manager/0.log" Oct 31 20:37:34.153819 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:34.153786 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" event={"ID":"b6dc6fd1-1e89-4330-bb79-881ad9f96585","Type":"ContainerStarted","Data":"272b60c5c22c74a84ae1019bf3cd4ac884676d79b93c368a87e312c3c511ca69"} Oct 31 20:37:34.154263 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:34.153838 6107 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" event={"ID":"b6dc6fd1-1e89-4330-bb79-881ad9f96585","Type":"ContainerStarted","Data":"7fe74b04bf1a2e6c893d19ad8f766f806524b275c457777c09cadc29fcdb228a"} Oct 31 20:37:34.154263 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:34.153876 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:34.260561 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:34.260495 6107 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" podStartSLOduration=2.260480632 podStartE2EDuration="2.260480632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 20:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:37:34.258988454 +0000 UTC m=+4825.892382364" watchObservedRunningTime="2026-10-31 20:37:34.260480632 +0000 UTC m=+4825.893874539" Oct 31 20:37:36.675614 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:36.675587 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a86c21-e504-455a-83d5-e68baf8191c3/alertmanager/0.log" Oct 31 20:37:37.062922 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:37.062895 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a86c21-e504-455a-83d5-e68baf8191c3/config-reloader/0.log" Oct 31 20:37:37.450051 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:37.449987 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a86c21-e504-455a-83d5-e68baf8191c3/kube-rbac-proxy-web/0.log" Oct 31 20:37:37.841679 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:37.841646 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a86c21-e504-455a-83d5-e68baf8191c3/kube-rbac-proxy/0.log" Oct 31 20:37:38.229660 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:38.229628 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a86c21-e504-455a-83d5-e68baf8191c3/kube-rbac-proxy-metric/0.log" Oct 31 20:37:38.616664 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:38.616579 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a86c21-e504-455a-83d5-e68baf8191c3/prom-label-proxy/0.log" Oct 31 20:37:39.005109 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:39.005082 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_45a86c21-e504-455a-83d5-e68baf8191c3/init-config-reloader/0.log" Oct 31 20:37:39.876652 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:39.876619 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hd95f_e1441c08-c76a-4954-80c9-8c1a0a04ce9f/kube-state-metrics/0.log" Oct 31 20:37:40.169128 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:40.169056 6107 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-kgmsz" Oct 31 20:37:40.256786 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:40.256759 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hd95f_e1441c08-c76a-4954-80c9-8c1a0a04ce9f/kube-rbac-proxy-main/0.log" Oct 31 20:37:40.640298 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:40.640268 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hd95f_e1441c08-c76a-4954-80c9-8c1a0a04ce9f/kube-rbac-proxy-self/0.log" Oct 31 20:37:41.804087 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:41.804056 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2hfw5_6d131de0-3a07-4d70-99e7-7aedb4648f31/node-exporter/0.log" Oct 31 20:37:42.195072 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:42.194996 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2hfw5_6d131de0-3a07-4d70-99e7-7aedb4648f31/kube-rbac-proxy/0.log" Oct 31 20:37:42.578799 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:42.578772 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2hfw5_6d131de0-3a07-4d70-99e7-7aedb4648f31/init-textfile/0.log" Oct 31 20:37:45.276491 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:45.276461 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-n7m2s_6310b145-4c78-4f58-91d0-be7558e3b1cc/kube-rbac-proxy-main/0.log" Oct 31 20:37:45.708308 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:45.708212 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-n7m2s_6310b145-4c78-4f58-91d0-be7558e3b1cc/kube-rbac-proxy-self/0.log" Oct 31 20:37:46.093313 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:46.093279 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-n7m2s_6310b145-4c78-4f58-91d0-be7558e3b1cc/openshift-state-metrics/0.log" Oct 31 20:37:46.489538 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:46.489507 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79735aa7-bcbb-4428-8082-1c41a0db7b46/prometheus/0.log" Oct 31 20:37:46.873216 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:46.873129 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79735aa7-bcbb-4428-8082-1c41a0db7b46/config-reloader/0.log" Oct 31 20:37:47.263508 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:47.263474 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79735aa7-bcbb-4428-8082-1c41a0db7b46/thanos-sidecar/0.log" Oct 31 20:37:47.649113 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:47.649023 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79735aa7-bcbb-4428-8082-1c41a0db7b46/kube-rbac-proxy-web/0.log" Oct 31 20:37:48.036895 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:48.036866 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79735aa7-bcbb-4428-8082-1c41a0db7b46/kube-rbac-proxy/0.log" Oct 31 20:37:48.422220 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:48.422139 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79735aa7-bcbb-4428-8082-1c41a0db7b46/kube-rbac-proxy-thanos/0.log" Oct 31 20:37:48.807115 ip-10-0-128-164 kubenswrapper[6107]: I1031 20:37:48.807085 6107 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79735aa7-bcbb-4428-8082-1c41a0db7b46/init-config-reloader/0.log" Oct 31 20:37:50.403127 ip-10-0-128-164 kubenswrapper[6107]: E1031 20:37:50.403090 6107 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zxkn8" containerName="extractor"